What are your programming "hunches" you haven't yet investigated?
155 points by WilhelmVonWeiner
155 points by WilhelmVonWeiner
Do other Lobsters have any hunches that certain aspects of problem could be better solved by doing x, or applying y, but haven't had the time to investigate them or the confidence to rubber duck them? Hunches often create room for interesting discussions or new ways to think about problems.
Starting a cross-platform GUI toolkit project from accessibility instead of rendering would be much more practical and would result in a complete toolkit earlier than other way around.
This wasn't our goal with EtoileUI, but it probably ended up being similar. We took a bunch of ideas from Taligent, which had some very nice abstractions for separating an abstract tree of views from how the views are actually rendered, and separating abstract tools from things like touch or click events. One of the things I'd hoped we'd be able to do (no one ever got around to trying before folks moved on from working on the project and it died) was also building text UIs from the same abstract view tree.
For things that aren't reliant on custom views, it would be so nice to have an application that could 'render' as voice, as a TUI, a local GUI, or a remote display via a web browser.
This passes a gut check just because accessibility is so much harder to add to an application when it's otherwise done than gradually from the beginning.
Fixing my ambiguous phrasing: accessibility is so much harder to add to an application in a final pass than gradually from the beginning.
This seems interesting. How does accessibility drive architecture? How might the architecture differ between the two approaches? (I know relatively little about accessibility or indeed frontend development except something something screen readers).
Not my project, but multiple ideas immediately come to mind, with the first being testing. Since the accessibility tree would be available early in the development cycle, there would likely be a large number of tests written to ensure that the accessibility tree matches what I expected. It also means that I would optimize this case to make writing these tests as easy as possible.
The downstream effect is that developers using my toolkit would also have easy access to tapping into the accessibility tree to test their own applications. I've seen very few GUI testing techniques that are truly useful, but linking in to the accessibility tree feels like the right level of abstraction to catch behaviour issues but not failing every test when the developer turns on dark mode.
Speaking of which, if accessibility was the primary target, then you could imagine the actual graphical bit just being a theme or stylesheet on top of it. This provides quite a bit of customisation opportunities to the application developer.
The testing bit also ties in nicely with scripting. Since our primary interface isn't mouse clicks at a specific X and Y coordinate, but a higher level accessibility primitive, it's simpler to control it via some AppleScript/Rexx/Lua. It's obviously possible to integrate these abilities after the fact, but going accessiblity first makes it hard to write yourself into the corner where it is difficult to add these abilities.
How does accessibility drive architecture?
You reify the description of the packing and relationships into explicit data structures from the beginning.
So, you have to describe the relationships between things explicitly in some data structure. Then you "pack" things and put that into another data structure. Then, you apply the events or interaction which gives you another data structure. Then, you use that to actually build the render artifacts and caches. Finally, you actually render.
The accessibility folks get to grab the data structures that still have semantic meaning before rendering strips it away. The rest of us get a UI that isn't restricted to the "main loop" and the ability to do actual parallel UI rendering.
The issue isn't "accessibility" so much as "blowing away the cruft of design decisions rooted in the 1990s". Memory and CPU are plentiful instead of scarce. GPUs are super powerful. Trees used to be good to share information but are stupid when chasing pointers is more expensive than recomputation. etc.
The problem is that a GUI toolkit is a massive (see IMGui) amount of code that really isn't very interesting to most programmers.
I've had this exact idea and have made a few attempts at assembling one. Ideally it would be an a11y-only "invisible component tree" library that could be used alongside any existing graphics-only GUI library, like IMGUI. The problem is that there are 3 separate accessibility libraries (for Windows, Mac, Linux) and they're all archaic and poorly documented. It might be more feasible now than it was before, with AI-assisted research; understanding these kinds of legacy APIs quickly is one of the uniquely good use cases for LLMs.
This is what AccessKit is, it is written in Rust but does have both C and Python bindings as well. I don't know if it has been integrated with imgui, but it has been integrated with the similar Rust gui lib Egui.
https://github.com/AccessKit/accesskit
One of the main authors Matt Campbell did a great talk at the RustWeek conference last year which is worth a watch and/or listen if you have time: https://www.youtube.com/watch?v=OI2TzJ6Sw10
Starting with accessibility has the possibility of improving other features as well: test automation, localization, and, in the context of the web, that gray area between applications and documents such as deep linking and summaries for unfurls and search engine listings. Most of the time, these are treated as independent chores to be tacked on to the end of application development. It’s interesting to imagine what application development might look like if the runtime treated these as fundamentals.
It's possible to save a dramatic amount of memory (close to half for programs with pointer heavy heaps) and also speed up programs by a modest amount (by way of more data fitting in cache) by using 32 bit pointers on 64 bit systems.
The Linux x32 ABI lets you do just this, but it's tragically underused and overlooked. It's disabled by default on Debian (though you can enable it with a boot flag) and not even compiled in on Arch Linux. Most software compiles just fine for x32, but there isn't much packaging for it so you do have to compile nearly everything yourself.
I experimented with deploying mastodon on x32 and it cut the app's memory usage from 650mb to 350mb. There's so much potential here, but the work is mostly of the thankless coordination and communication type and I don't have the time or motivation to push it forward myself.
Pointer Compression in V8, kind of takes this idea, but instead of just saving memory they also get an in-process sandbox mechanism.
Yeah Java does this too, and I expect most languages with JIT will have at least eyed it.
Yeah one of the promises of multiarch support was being able to have both amd64 and x32 binaries on the same system working together in harmony (at some cost to disk space). Most programs should be x32, apart from those rare things that can grow to multigigabyte address spaces. (It gets fun for things like script interpreters where a program written in Python or whatever might want to choose the x32 or amd64 version of the interpreter based on whether the program never needs more than a gigabyte.) Debian supports multiarch amd64+i386 but i386 has several performance disadvantages compared to x32: fewer, narrower registers, and weaker SIMD and fp. I wonder, if Debian produced an x32 version of the distro, could it use a more modern instruction set baseline than amd64 to eke out a little more performance :-)
The problem is the same as for 64-bit MIPS, which supported 3 ABIs:
If you had a fully n32 userspace, you saved memory. But if you had one n64 process, you ended up with two copies of a bunch of shared libraries being resident. And that can easily offset the savings from smaller pointers.
You also end up with multiple copies on the disk, and multiple copies to download security updates for. This is why Windows prefers a WoW model for 32-bit, where the '32-bit' libraries are tiny wrappers that proxy to the 64-bit ones, and why the Arm EC ABI on Windows exists.
I am not criticizing Debian often because I am a huge fan and longtime user, but basically every time I added multiarch on a system (ususally adding 32bit to existing 64, so not talking decades ago) I ran into oddities and edge cases and it's about the only time where I felt I needed to clean install instead of dist-upgrade later. I don't remember specifics, but it wasn't enjoyable (although the parts that were not involved continued to work, so the system wasn't bricked) and I also had the exact same experience on Ubuntu. Most (not all) of these tries were related to Steam and games but I think this shouldn't matter for the sake of this discussion because in the end it was just adding some standard packages.
I'm assuming you are specifically talking about AOT compiled code? Some GCs for the JVM already support compressed pointers.
I think infrastructure-as-code has the fundamentally wrong approach, where we define what we think meets our requirements, instead of where we define properties the system should meet to be considered a solution to the problem. Similarly to property-based testing (e.g. hypothesis) we should have property-based infrastructure: a specification that defines the minimum requirements for our applications. There are a lot of problems that arise from this way to approach, like how cloud resources are stateful and come with side-effects, or how the search space for solutions is incredibly large. But, and this is a big but - similar work has been done with FPGA placement (which is where I got the idea) and k8s scheduling, so it doesn't seem impossible... just very hard. That's why it's a hunch.
Also shove a logic language like Prolog or MiniKanren in there. I'm sure there's room for it
very interesting idea. something like
I wonder how you describe specifics. or if it even matters. do you also say that port must be 443? that comm should be ssl encrypted? by an auth that a generic browser trusts?
For a long time I've thought our current IaC is wrong, but my idea centers more on execution (k8s is incredibly complex), than a perspective shift in declaring how things exist.
This is how I think of maintaining a clean codebase. I have begun thinking that, for the first time (really), these principles might need to be articulated as part of the "code", for agents (and the occasional human maybe) to follow. For example:
The rules that senior developers have been following for years, not general principles but their specific manifestation in a particular codebase, should now be articulated, maintained, and enforced by process to keep agents on track. This tracks with the feeling that programming is having its level of abstraction lifted.
And as a follow up, this really got me thinking! Maybe a logic language for expressing those rules/guidelines/principles would be very powerful in this AI era.
Logic languages truly make the most sense for configuration. There's some prior history and recent experiments:
While I haven't had to deal with infrastructure-as-code in close to a decade (which was primarily through Terraform), I 100% agree with your idea. One of the most frustrating pieces to deal with was the stateful/side-effects aspect that you call out: the Terraform configuration nominally looked to be declarative, but in practice it suffered hard from the "warm start" problem. If the machine that managed your Terraform deployments lost its "deployment history" database (I apologize, I don't recall what the state file was called), the next run would inevitably tear down some resources and re-create them while destroying the (transient|permanent) state those resources held.
I think there's a bit of a two-sided problem there and it might actually be more solvable on the provider side: instead of the provider giving an imperative API for the declarative/property-based tool to look at through a paper towel tube, the provider itself could take these descriptive configurations as input and compute the resources to satisfy them on its own.
While I haven't had to deal with infrastructure-as-code in close to a decade
What magical place do you live in and how do I get there?
What magical place do you live in and how do I get there?
Embedded, which has its own unspeakable horrors :)
If the machine that managed your Terraform deployments lost its "deployment history" database (I apologize, I don't recall what the state file was called), the next run would inevitably tear down some resources and re-create them while destroying the (transient|permanent) state those resources held.
This is called the state or "statefile", but the best practice has long been to put this in object storage and take care not to lose it. If it can't find things, it won't delete existing things (unless maybe you have a bugged provider) but rather it will try to create things that already exist and it will usually fail with a 409 Conflict error or similar. There are plenty of reasons to dislike Terraform, but I think this one is mostly not a big problem.
the provider itself could take these descriptive configurations as input and compute the resources to satisfy them on its own.
I think you're very close to reinventing Kubernetes controllers, which I'm increasingly coming to believe are the better way to manage resources (Kubernetes still has its own issues, but I work in both Kubernetes and Terraform on a regular basis and Kubernetes controllers give me far fewer problems).
I think you're very close to reinventing Kubernetes controllers, which I'm increasingly coming to believe are the better way to manage resources
It has been a while since I've fought with K8s as well and I did have a relatively similar experience, but there too I was frequently surprised at how many things reacted not based on the actual present reality but on their own cached notion of reality
There are plenty of reasons to dislike Terraform, but I think this one is mostly not a big problem.
I disagree a little.
This is a large pain in the butt because Terraform has exhibited bugs like, if a single item in your hundreds of deployment items happens to hang instead of failing or succeeding, then you get a stuck Terraform deployment that doesn't write out its state file until it eventually times out unconditionally after something ridiculous like ~30 minutes. I've seen the AWS plugin do this when it tries to create x509 certs if anything in the process fails.
The other thing is that I think the statefile is totally necessary anyway so it isn't Terraform's fault. It has to store the ids of created resources because I think there's no other way to avoid leaking resources in AWS without risking accidentally deleting unrelated resources in the same account that don't belong to your deployment. Also some things like random_password need it.
Couldn't each plugin decide on a tagging scheme to distinguish things it created from unrelated things?
This is something that's been in the back of my mind recently: how to clean up cloud things reliably. I kind of want to create a scope or arena where I can create a bunch of resources, then tear it all down; wondering if tags are a good way to implement that.
Yyyyyyeah I think you could do it by tagging all the resources with a uuid for each deployment (e.g. "bob's dev setup" is one, "production" is another, "staging" is a third uuid...) and then you can safely assume that all resources tagged with that uuid are yours to keep or delete next time you do a deployment.
Isn't the k8s controller approach to IaC, basically what has been done in Crossplane? Or is it another thing? :)
Crossplane is one way to do IaC with Kubernetes controllers—it aims to be easier than writing your own controllers for different resources or assemblies of resources, but I haven’t used it yet.
This is called the state or "statefile", but the best practice has long been to put this in object storage and take care not to lose it
We used Terragrunt (beta?) in 2014-2016ish because terraform did not have this thing baked in back then to sync it to s3, so the person you were replying might very well just have missed this later-widespread practice by a year.
Agree on both counts.
I think keeping state is the best solution to "the removal problem". That is, if you define a resource in your infrastructure as code and then you remove it from the code, how should it be removed? Ansible/Puppet/etc. do not solve this. I see some other approaches, but state (for me in object storage) so far has worked well for me.
When I was beginning with Terraform, I thought state would be a headache. But it hasn't; setup was much easier than I thought, and then I haven't had troubles with it (yet).
(My main problem with Terraform is tedium, which I think segues nicely into the next point :)
Yeah, the idea of Kubernetes controllers is intriguing. You define a data type that represents a complex component (e.g. a database with replicas and backup processes) and the controller transforms that into lower-level components (e.g statefulsets, cronjobs, etc.). I'm not in love with the idea (esp. because I don't like the eventual consistency stuff in K8S, although I guess it's the way to go), but I really can't think of better approaches.
I think in the end, some bits of infra are stateful and you have to deal with it. So far, for personal stuff I try to split stateful bits and reduce them as much as possible; I think different approaches are better for both sides.
(And of course, NixOS-like systems are likely the way to go. I like that more than containers, although so far I've not seen an opening to apply NixOS-like approaches :(
the provider itself could take these descriptive configurations as input and compute the resources to satisfy them on its own
AWS CloudFormation is an example of this. And GCP had Deployment Manager, but I think they gave up and switched to Terraform.
CF makes sense in theory, but in practice I’ve found there are frequent surprises, arbitrary limitations, and lack of observability when things go wrong. It’s also extremely fragile, because it assumes its saved state is always correct, and has no way to resync with reality, nor any easy way to avoid accidentally modifying resources maintained by CF. You have to set up “drift detection” to catch that, and—in classic AWS style—though that’s an implied requirement for this design to work, AWS leaves you to implement it yourself. Once drift is detected, fixing it is a manual process. I guess the intention must be that you go full GitOps and lock down permissions so only the CI can change anything.
I don’t see any theoretical reason why CF couldn’t do a better job, though.
I feel like a chat system with a MUD-style interaction flow might do a better job of replicating the good aspects of in-office collaboration than something like Slack.
Couple (non-exhaustive) reasons:
interestingly, this is approximately how things worked in IRC 1.0. you could only be in one channel (each with a numerical ID) at once. incidentally, this is why there is a PRIVMSG command and no MSG command. I was curious about this a while ago, and looked at the old sources, and lo, there was indeed a MSG command, with one argument, that would send a message to your current channel. when named channels came into existence, PRIVMSG, already used for direct messaging, was reused to interact with them.
A friend runs a company where they use (or used) something like this! I think it was gather (scroll down to see an example "office" in 2D).
Yeah, gather.town looks nice. But I actually feel like it's maybe a bit too much! I don't think you need to have a whole 2D environment to walk around in - just doing it with text is probably enough to shift some of the social dynamics.
For broad Slack-level adoption you definitely need a spacial representation and integrated movement. Some people just fundamentally dont't feel comfortable with mixed mode inputs. Someone once saw me on screen share typing /gif lol into Slack and said "oh how could I do that?" — they saw the slash command, but assumed they would never do that themselves.
In '22 or '23 I attended an online conference using gather.town and IMHO it didn't pan out at all. But it was one of the "good in person" recommended conferences so maybe the crowd jsut didn't match.
I'm not saying it can't work, but it feels a little like solving a social problem with technology ;)
Some companies manage to have good and cool #offtopic channels that actually work, most don't. I still think it's about the mindset of the people. If you have irc fans who love to type a lot - great. If you have mostly people who want a video call for every small problem instead of typing 3 sentences... nothing will work.
I had a few good experiences with gather town. The thing that they did really nicely was make audio and video fade in gradually as you got close to people. This, combined with not bothering to do collision detection between avatars, means more people could easily gather around a small group of people having an interesting discussion and join in than they could in person. And it was easy to wander around the virtual environment and listen on the edge of groups before finding one that was interesting enough to join in.
The only place it didn't work well was for presentations. They had a hard limit of the number of people who could receive the same video feed (100? 150?) and there were more people than that, so the last people into the room didn't get to see the presentation, which was like a real conference only worse.
Yeah I think we had the video separate outside of the web app and the "presentation room" was just for chat related to that talk, that part worked.
I wish it had gone better and I'm not against trying again, but so far I have to live with the one meh experience.
Being able to see what chat channel your coworkers have "active" in Slack might be a simple way to implement this. I love the idea of improving "rich presence" for enterprise chat apps.
I thought something similar about video calls. Rather than break out rooms you could map yourself in a 2D space. If you wanted to join a conversation, you would just drag your icon closer to a cluster of other users. Then it would increase the volume proportionally to your distance to them. If you wanted to get a bit of chatter as background noise, slide yourself away. Not sure how that would work in practice though.
Users would be happier with the visual theming of their choice applied to their entire computing environment, and no application-specific theming or site-provided CSS, with an exception made only for video games where the theming is more entertainment than brand.
Just to clarify, you're saying that if someone reads Financial Times, NYT, The Onion etc., and they all used the same fonts & styles (based on what the user picked), the user would be happier overall?
A small but important fraction of users will likely be happy that they can make the font bigger, avoid low contrast, choose a more legible font for them, etc.
This is one of the most compelling aspects of the performance act that is Gemini, and this is why my personal website is Gemini-first and the CSS for the HTML version just sets 8 properties.
I don't think this is the right approach for everything, but I think it's the right approach for more things that most web developers think.
(Also, think about prose-y books. Who cares about any of the visual aspects [unless it's not to your liking, in case which... then I bet you'd wish you could change it]?)
You can already do this in Firefox's Reader Mode. I'd also note that epubs also already work like this!
In my opinion, Reader Mode demonstrates we need more of this!
Reader Mode is wonderful and a great piece of programming. However, it's a "hack", because it works around limitations (or rather, features) of HTML. I have found pages that should work on Reader Mode that don't. And more importantly, I think you can't put your browser on reader mode and even if you could, I don't think it would work well.
Gemini is a medium where you are always in reader mode, and in my opinion, it makes for a significantly different experience.
(Of course, you cannot cram the whole world in Gemini. For example, Lobsters cannot be done in Gemini. But then I think that Lobsters maybe should be email/NNTP.)
edit: and about epubs; epubs are also a second-class citizen in the world. It is excessively hard to make and read epubs. You need arcane knowledge outside some specific common cases.
(And also, random note: epubs IMHO don't work very well for example with computing content and probably other kinds of contents.)
Right, makes sense!
Sort of related, sort of unrelated, but I'd love to see browsers accept + render more file types / network protocols. I bemoan that no browser even attempts to render Markdown by default... or that yeah, other protocols like Gemini or IPFS are a no-op. I don't even know if it's possible to add support for Gemini to Firefox through the WebExtension API.
Unrelated, but related to me :D
My Gemini-first website is based on content negotiation implemented completely in Apache httpd (+ a thin Gemini reverse proxy on top).
So example.com/foo serves HTML if you Accept: text/html, but my Gemini reverse proxy requests gemtext and that's how it works.
Incidentally, if you use curl on most of the URLs on my website, because Apache httpd has this wonderful behavior of choosing the smallest file when it doesn't know what the client wants, you get the gemtext too. Which works much better on a terminal than HTML :D
(The bad exception is that I abused content negotiation for RSS; /index of my blog serves /index.html, /index.gmi and /index.rss. I use the HTML header thingie to point RSS software to /index.rss explicitly, but I thought it's more "correct" to have the RSS be an alternate view of /index. Because my /index.gmi is much more long than the RSS feed, because the RSS feed is limited to 10 entries but the HTML/gemtext versions contain all posts, curl mysite.example.com returns the RSS, so this beautiful behavior is hidden on the "most important" page of them all.)
Yeah. I mean, my favorite sites probably wouldn't be the reason it would be pleasing to me. But all the otherwise obnoxiously branded sites and apps, themed my way instead of according to their various stakeholders' vanity? That's peace.
I would love an experience where my browser opened everything in Reader mode (and that mode worked on every page/site).
It sounds like you're describing something closer to the old Mozilla user.css though.
I do use reader mode by default in Mobile Safari, and I switch it off for some sites (like this one). But yeah, a systemwide user stylesheet is appealing to me.
Firefox: Settings → Fonts → Advanced… → untick Allow pages to choose their own fonts, instead of your selections above.
It makes the web so much better. Flipped it as a one-week experiment six years ago. After the week, turned custom fonts on again for a few days just to confirm my feeling, and yep, hated it.
Seeing everything in the one font does make me more sensitive to letter-spacing abuse. Strong opinion: for body text and headings you should never set non-zero letter-spacing. Display (meaning: artistic presentation) and all-caps are the only times it’s acceptable. I’ve contemplated adding a * { letter-spacing: unset !important } user stylesheet, which might damage a few things but fix a lot more.
I wonder how this impacts pattern recognition. Generally the styling of an app/site helps me identify it quicker (whether in alt-tab or regular usage), and it seems useful to be able to build different patterns for different apps.
You could allow people to do things such as pick colors for domains, etc. There are likely many alternatives out there.
And all of this could be optional. Websites could make it easy to use "their theme", no theme, or the user's theme.
This was a goal of CSS from the beginning - it never materialized. And I imagine over engineered class-based css frameworks would make it very difficult today.
Ages ago I did some prototyping for a model for exception handling where you used the same register for normal and exception returns, and set the carry flag to differentiate these. Each call became a call followed by a branch-on-carry to get to the landing pad. The branch-on-carry is statically predicted not take and has the same overhead as a nop on most cores. If a function just rethrows, that branches to your epilog (which needs to preserve the carry flag, but is otherwise unmodified).
I hypothesise that this will give a fast and small way of doing exception handling - smaller than existing unwind tables and comparable performance in the non-throwing case (much faster in the throwing case).
But I've never got around to building a proper implementation that could be used for more than microbenchmarks.
I have a similar idea, but for error handling, not exceptions. I envision each function having two return types---a "happy path" and an "unhappy path," using the carry bit to flag what path is returning. If a function returns an error, and there wasn't explicit code to handle the error return, the compiler would then add code to return an error. The result looks like an exception, but hopefully with a lighter syntax when writing the code.
It is more important to design a project's culture than its codebase, and more important to design its change control process than any feature. The general principle is the common saying that a little slope makes up for a lot of Y intercept. The wrinkle is that projects won't be viable without enough up-front value and promise to attract a community. We mostly talk about the less important thing because it's a precondition of the more important thing.
Yes, but I think an ever harder and more important problem (and more often ignored) is "going in the right direction". (However, probably a good project culture is a culture that helps you go in the right direction.)
But what is true is that often we only think about code and think other stuff is less important.
Nice link feature (text selection)
It's a browser feature so you can use it everywhere!
https://lobste.rs/s/gns27z/what_are_your_programming_hunches_you#:~:text=Nice%20link%20feature
After commenting, I realized there was no js in the website, so I thought it might be so. I should have checked before commenting!
While I agree, there are also many ways a project's culture is enforced via code based on tooling, like repo merge controls and commit hooks and CI and linting etc. I think it's all one big continuous system from the actual code to the org culture.
Combining the terminal and the shell would lead to significant UI improvements. Imagine being able see all the active jobs in a side pane, for instance -- a small thing but the small things might add up.
We could then extend the idea further: what if command-line tools output rich text documents, something better than CSI escape sequences? We could have proper tables, headers, layouts, etc; with the right format, that document could be easily parsed and manipulated that doesn't sacrifice composability. (I'm thinking of something like HTML's id tag. For example, if a rich-text version of ls output a table of all the entries in a directory, it might tag it with the entries label, and then that table could be extracted into JSON with something like rich-ls | extract --json entries. Or maybe just something like rich-ls | .entries | into json. If we're combining the shell we may as well reinvent the shell language as well, I guess.)
I am thinking about this for many years now. I would also prefer to think about jobs output being scrollable independently, so by default everything runs in parallel. The closest thing that I still didn't try out is cat9 from Arcan project. Though I am unsure if terminal and shell is a single process, I understand that there is some lua sparkled in-between.
I would also like to have "debug" pipes. So when doing a pipeline every part is something you can snoop or pause or change. You would basically see the buffer and see what was already consumed and you could change what wasn't yet consumed. It could also show processing speed like pv.
I would like to treat file descriptors as first class objects and also see their state in some pane if I desire.
For some time I also wonder how it would all work with a real web desktop. I know about "web desktops", but they either work as full desktops, but written in JS/HTML (so OS shaped web app) or something tied really to a single process on a server somewhere. I would like to think about it more like what Display PostScript was supposed to be about, but with "current" web technology. So apps would be regular processes that talk HTML/JS with the display server, but with some APIs for this to make sense. Then remoting would be really just going to a web server. I would like for it to support opening an app window separately as a new tab. But this strays from the main issue too much :)
cat9 is indeed very close to what I was thinking of. I had a hunch I had seen something similar before, but couldn't remember for sure.
I love the idea of "inspecting pipes". It's one of those things which, when you think about it, is surprising that we don't already have. I'm not sure what you mean by file descriptors as first class objects. What file descriptors are just open all the time (besides stdin and friends)? Bit confused there.
It also reminds me of a bunch of other things I'd want in a shell+terminal combo.
cat9 author is doing in that article. And sure, it can be customized with a different command to, say, track the git status in the current directory. But either way, it should be rich text: I should be able to click on any directory to cd into there, instead of banging out 20 cd invocations like a caveman. Something like broot would be ideal.$HOME?), and even things like the git status at the time.cat9 article. We can take it a step further: when typing out a positional argument, the GUI should highlight the text I'm typing and have a tiiiny hint at the top with the name of that argument. So if I'm typing out ln -s fo_, which my cursor at _, that word will be highlighted and a little "TARGET" will be visible above it.I like to use process substitution a lot: >(grep whatever) <(diff this that). I would like to be able to play with more things like that. This would give ability to do more complex pipelines. With those "inspecting" pipes it could be possible to change the pipeline as it is running if one wants to. It would be also easy to give out temporary file as memfd. All this with easy sending of file descriptors via SCM_RIGHTS.
I like your ideas and I would add easy sandboxing. Docker doesn't offer fully what I am after. I think about two basic types of sandboxing: separate system (like regular docker; yes, docker is not really a sandbox) and restricted local environment. For the restricted local environment I would like to be able to do two things: 1. hide personal data, leave the rest mostly as is (read-only), 2. rw overlay over whole system (the thing can change things, but changes are visible only in the sandbox).
Cat9 looks to be a very interesting and advanced way of using the terminal. Unfortunately it looks so hard to install, especially on macOS. Thanks for sharing anyway.
I’m aware of nushell and powershell, but sadly it falls far short of what I'm looking for since it doesn’t control the graphical interface. They are “just” shells. What good are tables if you can’t sort by a column, or right-click them to assign them to a variable for later?
The second half of this, with talk of tables and parsing and conversions, is present pretty well within PowerShell due to it being object based.
You can, for example, run a command like "Get-ChildItem | ConvertTo-Json" and receive your current directory in json format. Similarly, you can write to customized formats and built-in tables, convert in and out of things like json and html, and more.
Yeah, I've use(d) PowerShell extensively back when I was on Windows, which is where the inspiration for that comes from. Admittedly I haven't used nu-shell, though my impression is that it's like PowerShell and has all the nice/"rich" commands builtin, while I'd prefer that a standard interface be defined for any program to output objects and documents.
Still miss PowerShell though. The builtin command-line argument parser was lovely.
A great presentation in the same vein https://www.destroyallsoftware.com/talks/a-whole-new-world
I love that talk. Gary threw down the gauntlet in IDE design, and I picked it up.
He called his (fictional) product Aneditor; I call my real tool Paneditor as a doff of my hat to him, and to my dearly-loved cat Pan who passed away while I was building it
I added a similar idea to my idea document recently. You could have a standardized interface that signals that a program's output is HTML (or another format) or that the program binds to a port and starts a web server. top would be a small webapp.
(You then could do web browsing as tabs in your terminal too :)
(I'm quite surprised when I search for this I never find anything...)
A smaller scoped version of this could be a TUI shell. Most shells don’t take over the terminal the way vim or htop does, but they could, and it would make it easier to implement things like foldable history or a background jobs sidebar. You could implement a really different shell experience even without replacing the terminal.
I started using kitty terminal recently and realized that mouse and keybinding possibilities for TUIs have gotten much better—mouse hover events in a TUI are possible, for example.
By keeping it in the terminal layer you also gain a lot of composability, like ssh and tmux.
That would be mine as well a "GUI Shell". I started tinkering with this and came to the conclusion that I'd like to have sh-compatibility (maybe oils OSH can help) and need a full featured terminal emulator (libghostty) for running commands/pipelines. But instead of a linear wall of text, I would use a GUI to model the interactions:
All of this is a little fuzzy, but I'd like a simple UI, with access to technicalities when needed.
Combining the terminal and the shell would lead to significant UI improvements.
Warp tried to do this, I think, but the implementation was so shoddy that it was just outright worse than split terminal+shells we're all used to.
Can you expand on this? I still use Warp, and I'm not clear what you're referring to.
I haven't touched it in a few years, but when I tried to use it last, I found that it broke pretty much every TUI program I tried to use it with.
Looking it up again now, it seems its done a major pivot to marketing itself as a front-end for AI tools. So I guess that ship has sailed.
Those are unrelated, I think.
I use Warp as my daily driver, and it handles my TUIs just fine. My only real gripe is you still can't add custom keybindings. They handle the common ones (Ctrl+r for history, e.g.), and support a lot of tools' keybindings, but you can't override/add your own by default.
Speaking of capabilities, I have a hunch that you can build a very nice capability system today as a (mostly) custom userland for the Linux kernel.
Linux already allows you to remove ambient authority (via namespaces and/or landlock), and it allows you to represent all kinds of capabilities as file descriptors. Also, you can pass those around between processes via e.g. SCM_RIGHTS.
I also think that many existing server-type programs can be ported to such an environment with a medium amount of effort, and that parts of the groundwork have already been laid by the systemd ecosystem. For example, many things log to stdout/stderr now (which are inherited capabilities) instead of writing to /var/log/whatever via ambient authority. Some programs also accept pre-opened sockets as fds for socket activation so they don't have to (have permission to) bind to addresses/ports themselves.
I have a hunch that you can build a very nice capability system today as a (mostly) custom userland for the Linux kernel.
I've also been thinking along similar lines recently. One could build a very nice non-POSIX OS on top of the Linux kernel's primitives (while maintaining the ability to run a normal POSIX-y environment in a container).
If you ever want to brainstorm or collaborate, feel free to reach out.
I wonder if you could build a general purpose stdlib and basic ecosystem that compiles to a small embeddable VM to make them portable to basically anything.
This is what the CLR purported to be: a common cross-language runtime. But .net is too closely fitted to C#; if it had been more language-agnostic there could have been (for instance) less of a compatibility break between Visual Basic 6 and VB.NET. It ended up being roughly equivalent to Java as a host for second-tier languages: not terrible, but second-tier languages like VB / F# / Scala / Clojure have a strong taint of their runtime’s primary language.
The problem is that a very large part of a programming language’s style and character comes from its runtime and standard library. If a programmer is going to the effort of creating a new language then surely they want its libraries to fit comfortably with its notation. For example, Clojure has its own distinctive collection types and iteration/traversal combinators, quite different from Java’s; I guess Clojure benefits from Java’s ecosystem so it can be batteries-included without having to recreate all the batteries. But I’m not sure if that peripheral stuff is the kind of “basic ecosystem” you have in mind.
I previously commented on C’s position as the lingua franca of FFIs. It’s much harder for non-JVM/CLR languages to make cross-language calls because the native FFI is so impoverished at the ABI level and requires most of a C compiler at the source level. Many things would be easier if there were a more language-agnostic IDL that could make it less horrible to link to native code libraries.
Project Cool (that became C#) was a clean-room implementation of Java, and the CLR was a clean-room implementation of a JVM -- all to dodge Sun's litigiousness. The CLR is only focused on cross-language in some theoretical way: It works fine as long as the language is (or looks just like) C#. It turns out that language semantics are shockingly different from each other, and it's hard to have an efficient "common language runtime" with a fixed instruction set that's any higher level than machine code. What LLVM and the Sun Labs projects Graal / Truffle showed is that it's possible, but you need more flexibility (i.e. something that goes outside of the "common" part). And maybe WASM's evolution is heading towards that same conclusion, like with the pluggable GC concepts.
The CLR is only focused on cross-language in some theoretical way: It works fine as long as the language is (or looks just like) C#.
Yeah, that’s why I wrote “purported” :-)
I have some half-remembered hearsay that the MSIL -> CIL change was tied up with the “standardization” of the CLR. Microsoft was keen to make their tech appear to be less proprietary to avoid another round with the competition regulators. But it isn’t clear to me how much the “common” stuff was driven by Microsoft or by the rest of the committee. I think HP might have been enthusiastic about it?
But Microsoft really did work on making the CLR a multi-language platform, eg there’s C++/CLI, and the dynamic language runtime that supports VB and IronPerl / IronPython / IronRuby. Tho there were far too many caveats for the latter to become popular, and C++ had (has?) proprietary “managed” language extensions.
Graal / Truffle has become much better at making dynamic languages go fast (with caveats about warmup time and memory usage) but no-one’s rushing to use a platform with complicated licensing by Oracle.
All good points, and also it's important to remember that lots of people with different goals were involved all along the way, so it's not a singular, unchanging vision over time ... Anders for example wasn't interested in doing a clean room Java.
If a programmer is going to the effort of creating a new language then surely they want its libraries to fit comfortably with its notation. [...] But I’m not sure if that peripheral stuff is the kind of “basic ecosystem” you have in mind.
Not OP, but from my point of view, there's a quite a waste of effort trying to port algorithms in the standard library across all the programming languages. Could it not have been O(m + n) than O(mn) effort, similar to LSP? Like, how differently could you implement per paradigm?: floating-point to string conversion and vice-versa, math functions, string functions and regular expressions, various container data structures and algorithms on them, date and time handling, parsing and producing various data formats (JSON, XML etc.), interface with operating systems, networking, etc.
The matter with ecosystem is quite concerning, for making it difficult for new programming languages to catch on, should they not target an existing language ecosystem or have a huge community or corporate backing to bootstrap the ecosystem in the first place.
Regarding language-agnostic IDL, there's one that I'm aware of: WebAssembly Interface Type.
Just off the top of my head, Haskell has five different string types (I.e. Text, Text.Lazy, Bytestring, Bytestring.Lazy, and, of course, String). None of these are the null delimited string or pascal prefixed string that I would expect to receive from the floating point to text function of the universal runtime. Even if they ringtone did return Haskell’s accursed String type, it’s not returning the other four, so those functions still need writing.
I mention this because I had the same thought as you, but slowly came to appreciate that new languages usually exist because the creator wants to do something differently from the old languages. Whatever that new thing is, it will break some assumptions that exists in the universal runtime.
Yep. Every language high-level enough to be interesting also has to bake in assumptions about how data works, including data like functions, closures and coroutines. COM is also an example. WASM is the reaction to that sort of thing, where you try to make your VM low level enough that you don't have many assumptions about how data works. This seeeeeems to be mostly working as a universal runtime? Better than Java or .NET has, at least.
Hmmm, I wonder if we converged on JSON for RPC because JSON is about as dumb as possible and assumes very little?
For example, Clojure has its own distinctive collection types and iteration/traversal combinators, quite different from Java’s; I guess Clojure benefits from Java’s ecosystem so it can be batteries-included without having to recreate all the batteries. But I’m not sure if that peripheral stuff is the kind of “basic ecosystem” you have in mind.
I'm thinking along the lines of libraries that handle dealing with HTTP (sans IO), compression, encryption, or date parsing: pure functions that have a small surface area but might do a lot of work. Things like iterators are probably too core and opinionated to be farmed out.
It's crazy hard to bootstrap new programming languages. I wonder if you could build a general purpose stdlib and basic ecosystem that compiles to a small embeddable VM to make them portable to basically anything.
I had a similar idea, some levels of abstraction higher. As long as things don't go beyond Turing-completeness, it'd always be possible to translate programs from one language L₁ to another Turing-complete one L₂; in the worst case, you ship an L₁ interpreter written in L₂ along with the program. In practice, languages are similar enough that we don't need to go so far, consider the number of languages that transpile to JS or C, for example. There's also this monstrosity of an example of what's possible: a PHP and a Python implementation surgically stiched together, that allow arbitrary nesting of code written in each into the other. The closest we have come so far in this regard is WebAssembly Component Model. There's quite some research been ongoing around this matter: DimSum, Pyrosome, Program Logics à la Carte, Compilation as multi-language semantics and so on.
I still think the KDEv3 reusable shared components model is good but it was stymied by OOP and the window management of the era. Single window interfaces were more space efficient on relatively lower resolutolution displays.
With tiling window management and modern high-level IPC mechanisms and process isolation mechanisms I think making GUI applications split their UI areas into separate windows managed by separate collaborating programs would go a long way toward unlocking the value of software. So much of an application UI is built around constructing a tree of possible visual presentation states that correspond to anticipated use cases, biased toward the most common. That makes it awkward to use a lot of software for unanticipated use cases, which leads to a lot of re-inplementation of the core data manipulations, IO, etc.
And why do we do it? To avoid an unorganized mess of dialog windows each cluttered with their own title bars, etc. I think further deferring to more opinionated window managers would be a better model.
Microkernels are king. Monolithic kernels run way too much in kernel-space and are subject to stability and security problems due to their large attack surface. Monolithic kernels were a mistake, one that we will need to fix going forward. I haven't fully investigated this hunch, but I have nearly a decade of experience with microkernels in production and the stability is unmatched. Even a bad driver is no problem; since it runs in user-space it can just restart when it crashes. Microkernels can have a bit poorer performance than monolithic kernels though.
Another related hunch: POSIX contains way too much legacy garbage (e.g. signals, fork, thread-unaware functions). Real OS innovation will require ditching POSIX and building on a more reasonable foundation. That one is probably never going to manifest because virtually everything (including compilers themselves) is built for POSIX.
This is part of the reason we did CHERIoT RTOS: it's easier to do innovative things in embedded because the OS dependencies for embedded code are much smaller / nonexistent, and we could use it to showcase ideas that would be possible on bigger systems but would require a lot more developer effort to implement.
I have nearly a decade of experience with microkernels in production and the stability is unmatched
Could you tell us more about this? I'm living in a Linux monoculture right now, so I would love to hear about people running production systems on a different foundation... especially if it's something exotic like a microkernel.
I've been using QNX. It's a microkernel (the entire kernel is only about ~50,000–70,000 lines of C code) that is POSIX-compliant. All IPC is either through message passing or pulses which are essentially tiny 40-bit "fire-and-forget" messages. Because it's POSIX-compliant, a lot of Linux software compiles and runs with little to no modifications. We ran this on embedded devices, but for unit tests, a QNX system was also integrated in our build server infrastructure. Basically everything except for scheduling and the management of the file system tree is in user-space. All drivers, like display drivers, even hard disk drivers are in user-space. They can crash and you can just restart them, no harm done. Kernel panics are simply unheard of, it's not a thing. The only way you can get a system to hang is when you do a "while(1) {}" at 255 priority for every CPU core, because the scheduler does exactly what you tell it to (supporting even real-time scheduling), and this will lead to starvation of the Ethernet driver, making the device unreachable. QNX is the best operating system I've ever worked with, by far. Going back to Linux and working with drivers in the Linux kernel will cause you physical pain. It's a shame that QNX is closed-source and that its ecosystem lacks many drivers and other software ports.
I'm not too familiar with QNX, but IIRC it's a microkernel that's also fast. Is it documented sufficiently that another implementation with the good properties would be possible (in a different language? Rust)? How does it compare to e.g. seL4? I would also like to see capabilities as a core primitive, since security is central to almost everything these days. But at what cost to performance?
QNX used to be open source. This stopped at around version 6.4 (many things seen in the OpenQNX source code are still exactly the same in modern kernel versions). QNX has evolved a bit since then, though. A Rust port would be awesome.
It's been interesting to watch the Windows evolution spiral of "let's build a microkernel (NT)" -> "it's too slow, make it more monolithic" -> "it's too insecure, split the OS into multiple VMs (VBS)" -> "containers are too weak for the cloud, use VMs". We now have a microkernel again, except that it's called a hypervisor a has a bit more overhead.
I wonder how far off "virtualization is too slow, migrate part of the kernel to the hypervisor" are we...
Performance is a concern, yes. Microkernels perform just a bit worse because of the stricter abstraction layers. I believe it's worth it for most use-cases, just like it's worth upgrading from DOS to modern systems even though that means you now have context switches between kernel and user-space, costing you performance.
What do you mean by "Real OS innovation"? Do MirageOS or Fuchsia count?
Yes, they count. I have to admit, I haven't studied them sufficiently to judge how good and innovative their design is. Also, adoption is basically zero. But there are indeed some developments in the OS space, it's just that they progress at a glacial pace and don't gain adoption.
Fun fact: the Fuschia project was started and led by Chris McKillop, who started his career at QNX. I used to use his Photon MP3 player and IRC client back in the day when running QNX as a desktop OS was viable.
Nintendo has been using in-house microkernels for their game systems since Cafe OS on the Wii U and Horizon on the 3DS.
This is also my take. Been writing an OS for a long time now, I still hold this (apparently unpopular) view.
I have a strong hunch that we're so close to a better UI/UX/CX of spreadsheets, with only a few changes needed:
Yes, yes, a lot of this can be achieve with products like AirTable or Notion or some micro-app of Sqlite + a few scripts. But I don't think anyone's brought it all together into a program that prioritizes user control + user value + user experience. Most of the examples I've found that come close have prioritized other aspects (profit, some specific use-case rather than a flexible one, etc), so haven't yet nailed it. I'm not even sure that I could do it if I had time.
Sounds like Lotus Improv in some ways. Ironically, Improv wasn’t great at the kind of improvisational development process that spreadsheet users are accustomed to (it benefits from more up-front design and planning) so it wasn’t successful. In many ways spreadsheets allow non-programmers to do programming without the cusomary bureaucracy of official IT programming; the kinds of features that would make spreadsheets more palatable to programmers would make spreadsheets less palatable to their users. Especially things like type checking that improve correctness at the expense of interactivity and immediate answers. Which is not to say that typed spreadsheets can’t be popular, but rather the kinds of type systems and how they present their feedback to the user need to be quite different from what programmers are familiar with. It’s crucial that a type system should be helpful, eg narrowing down which functions are relevant, rather than presenting obstacles to the user that must be dealt with in a way the computer demands rather than in an order that matches what the user is currently thinking about.
Totally great point about static typing. I'd envision this hunch of an idea to use the static typing to warn users of mistakes in a helpful way or like you said narrow the list of relevant functions. Finding the right balance seems tricky and is probably part of the reason we don't have this Spreadsheet Of The Future quite yet.
I think you're describing Microsoft Access, circa 1998.
It was a terrible database, but a really remarkably good spreadsheet.
Yes! I played a bit with embedding SQL in Markdown. (A Markdown table gets turned into a table; an SQL code block runs a query on the Markdown tables and creates an HTML table with the output.)
I thought Notion and Coda would herald a new wave of innovation in this area, but it barely went forward :(
This reminds me a bit of QGIS and how spreadsheets are tied to it. For the unfamiliar for map data (features: points, lines, areas) you can have some additional data. So for every layer you basically get a spreadsheet - rows are features and columns may have different data types (i.e. name, administrative division, population etc.). QGIS gives a simple form builder based on the spreadsheet for data entry. There is also some Python scripting you can do with all that to some extent (I used only simplest things like getting feature area).
At the same time some GIS formats are basically SQLite databases. There seem to be a massive overlap.
Yeah, I think it'd be very reasonable to store this improved-spreadsheet file as a sqlite db instead of some xml-based file (like xlsx) or on a server (like google sheets, air table, etc).
I'm going to have to look at QGIS to get a sense of the UI/UX of the flow too, that sounds neat!
The form builder is quite neat. I didn't use it for overly complex things so I don't know if it won't crumble under weight, but it worked good for my use. The spreadsheet scripting/formula part is something that I used less, though I think it is less exposed and more rough around the edges.
I think that there might be another alternative fundamental architecture for declarative web frameworks without the virtual DOM, based off of the incremental lambda calculus. A web UI, in the abstract, is a function from state to a DOM tree. Rerendering the whole tree every time state changes isn’t feasible, so the virtual DOM does diffing to only change the portions of the output which need to change. The incremental lambda calculus shows that it’s possible to take the derivative of arbitrary functions. The result is a new function that, given a change to the input of the original function, produces the change of the function’s output. Applied to UIs, this would mean that the change in the DOM could directly be computed based off of the state-changing events.
I think a straightforward implementation of this would probably be pretty tractable and highly inefficient in many cases. I’ve wanted to explore it for a while and see what potential the idea has but I’ve never found the spare cycles for it.
Rather than virtual DOM based frameworks like React, something like Solid is a little closer to this, where reactive signals are used to propagate changes to the DOM at the end of the day. (I think there are others than Solid doing this approach too)
I’m aware of svelte, which adds reactivity via a compiler, but I hadn’t heard of solid. I’ll have to look into this. I suspect that these production-ready frameworks will have made the tactical decisions needed to make this performant and ergonomic, and that the “haha calculus goes brrr” approach that I want to take will have a lot of pitfalls, but I’m personally more motivated by satisfying my curiosity and maybe writing a blog post than I am by making something useful here. I’m not even a frontend developer, so definitely not the best person to try to make a practical solution.
It’s been a while since i heard a datastar author explain it but it sounded like they do a similar approach feeding the UI with SSE and resolving the end state of the dom using the idiomorph library. Not an incremental calculus approach but still pretty cool.
This sounds a lot like Elm to me. Iirc they can mostly avoid using vdom because their state tracking system can identify exactly what nodes need to be updated for any given reactive update.
I think this is exactly what incr_dom is.
It’s related but different. (I went through all of the Jane Street incremental stuff a while ago). IIRC, their incrementalism approach is more akin to Excel-style dirty tracking, in which subsections of the computation are re-run as necessary rather than being based on function differentiation. They also still use a virtual DOM, though I believe one of their blog posts suggested that it might be possible to get by without the virtual DOM.
Although it doesn't leverage ILC, @thi.ng/rdom (under Karsten Schmidt's umbrella) is a way to write reactive, VDOM-less UI/DOM components with an async lifecycle. I used it in a small side project and liked it. It might give you some ideas and inspiration if nothing else.
what do you think of xilem? https://raphlinus.github.io/rust/gui/2022/05/07/ui-architecture.html
I believe that certain testing practices are incredibly effective at finding bugs for certain kinds of software, to the point that they can plausibly find all the bugs and act as correctness oracle. I have personally witnessed this for a datastructure library I was writing (store), where we use model-based testing using the Monolith library. It is incredible: if I wonder about why a given check needs to be done somewhere, I disable the check, rerun the testsuite, and in a second or so it shows me a minimal repro case of what goes wrong without that check. The test is so thorough that is serves as a tool to explain the code. (This is not just random-testing compared to unit tests; my previous usage of property-based testing never felt as good as this.)
Of course this can only work in certain problem domains, it would be too good to be true otherwise. For data-structure libraries I believe this works because there is a small and well-designed API, and testing by generating chain of API calls exercises the logic thoroughly. But I believe that other domains have their own "incredibly effective" testing approach (we hear a lot about "deterministic simulation testing" for distributed systems, maybe it is related? differential testing also comes to mind, it is criminally effective for compilers for example; I don't have experience to know if they feel like correctness oracles in practice), some which are known by domain experts and some which are not. My intuition is that this is related to the question of whether the code "has a specification" (that is shorter, more declarative than the code itself).
I believe that these "incredible testing" methodologies are not as well-known as it should, and that it is worth investing a lot more effort than we typically do into designing an "incredibly" testing strategy for any software project.
I have a hunch that PIC-like languages (such as Pikchr) can be excellent for all kinds of diagramming if they had a Drawio-like WYSWIG interface for precise manual control and broader audience. They would still emit reproducible diagrams-as-code of course. This hunch boils down to my frustration that I have to choose between:
SVG kinda could work maybe, but it's just a little too low-level, there isn't a concept of "move this thing 1cm away, make this arrow dashed". I suppose some SVG editors could be extended to make it work, but Pikchr already implements such nice abstractions on top of SVG, there just isn't a WYSWIG interface. My hunch is that it shouldn't be super hard to map to a GUI.
Have you heard of D2? They have a GUI frontend similar to Drawio that works with their language. Granted, it costs a subscription but I think it fits your description of what you're looking for.
That being said I kind of moved to just using Drawio for all my diagrams? I love the ability to write some code and have it render a graph but the control in Drawio allows me to make things look a bit better.
Totally agree. They definitely would also benefit from having a chat interface in the mix, so you could actually say, for example, "Yes, but I'd rather boxes foo, bar and blam be horizontal, while the w, x, y and z are horizontal. Also, make the service boxes use the gear icon."
Interesting.
What I usually find myself wanting is something CLOSE to this, but not quite identical.
I often want to generate diagrams from data, and I usually want to create a reusable pipeline so the diagrams can be regenerated when the data changes. That usually leads to using things like Graphviz but, (as you point out) that means I get very little control over the appearance, and often I want that.
What I wish for is some kind of a layout engine (actually, several layout engines that I choose between) that can be layered with "hints", AND a GUI display where the "hints" can be entered through an interactive view.
I'm daydreaming of a nix inspired system for scientific work, fully reproducible and tracked.
The big difference would be that instead of the nodes forming a Merkel tree of inputs, each node would isolate its downstreams from upstream changes ( as long as its output doesn't change).
Essentially a node has an input "name" (hash of hashes), and an output hash, and after resolving everything we know the complete mapping and have the requested output nodes on disk.
The idea is fairly straight forward, but the state space will require quite some engineering once you consider things like "runtime dependencies", remote oracles (that can tell you the output hash given an input hash), download from cache / oracle vs build decisions, etc.
And then you need a user friendly way to define the graphs :).
I have a hunch that there exists a precise language or interface for manipulating programs that would make us think of coding via agents as extremely crude.
To explain what I mean - pyastgrep is a precise language for locating syntax elements in code - i.e. the reading/search side of program manipulation. But it isn't brilliant for usability (you have to write XPath expressions), and it lacks anything for modifying code. There are lots of other efforts, like libCST, for the modification side. However they are relatively painful for simple refactoring tasks like "rename" or "extract method" etc. - in short, for small tasks they are harder than the more manual approaches, and for larger tasks coding agents are now what people turn to. My hunch is that there is something better, where you can both precisely and concisely express exactly what you want to do to the source code.
IntelliJ and other JetBrains IDEs have quite powerful structural search and replace which seems somewhat similar.
Decades ago, I remember someone trying a similar project. Their framework was a set of programs that could roundtrip other languages into lisp style sexprs. The procedure was then:
I remember that there were two big complaints at the time. The first was that people didn't want to write Lisp. The second was that the language conversion programs were fairly buggy. They did well in the general case, but frequently failed in the weird corner cases of the language. I'm now wondering if something like treesitter could solve the latter issue. I feel there's a Turing award for anyone who fixes the former.
Like Coccinelle semantic patches tho I guess you are hoping for something easier to use than Coccinelle…
Like, lisp and structural editing? You can arbitrarily change the source code. Indeed, you can change code behavior without modifying it. Common Lisp has good tools here. Image-based development is a whole big thing.
In some ways similar, but structural editing still doesn't capture many of the slightly higher level patterns I'm thinking of. Lisp is definitely going to be easier other languages to automate modifications, I can't comment on the tools in Common Lisp.
Have you read about Refactoring Browsers ?
They are roughly regular expression search and replace on the abstract syntax tree.
You then convert that modified AST back into the source code.
The closest thing we have now is tree-sitter, but it is sort of getting closer!
I strongly agree with your thesis that macros on the AST is a more powerful and precise approach for solving "the boilerplate problem".
That’s actually what the intellij editors are. Under the hood they work with ASTs rather than with the text files.
Just a small thing that I'm investigating now, but I thought it should/could be easier to write WebAssembly more or less directly than to target it with another language and jumping through hoops trying to convince the compiler to generate acceptable code.
WAT is s-expressions, so one could add some Lispy bits on top, including macros. I prototyped that and thought it would be fun to integrate SMT-LIB (also s-expressions) as annotations (using WAT annotation proposal.) "Codegen" is the s-expression writer dumping WAT into wat2wasm, which conveniently enough supports annotations (by ignoring them.)
Next will be allocator, macros to generate ADTs and records (supported by Z3 datatypes) and maybe an optional ML-esque syntax sugar.
Years ago, when WebAssembly was still relatively new to the web platform, I wrote a boids simulation manually in WAT. I didn't find it particularly difficult, although it should be noted that it's a pretty ideal problem for writing directly in some assembly language: small, self-contained, well-specified, and no string manipulation ;)
In fact, since the boids lived in buffers of fixed and statically determinable size it didn't require any memory management at all. This meant I didn't need to ship an allocator with the binary. When attempting to produce a similar demo using Rust compiled to WebAssembly, I struggled to convince the toolchain that an allocator was unnecessary. I think most of the overhead of using another language compiled to wasm is pretty negligible, but if you have a sufficiently well-scoped problem that you can avoid shipping an allocator, this might be worth doing.
One thing I didn't get around to testing with Rust is if it's possible to make a tiny leaking allocator, such that when you know you have bounded memory needs you can still satisfy the toolchain while not shipping a full allocator. I don't know what the state of Rust's wasm support is now. wee_alloc is still recommended by the Rust wasm book despite being archived, unmaintained, and having a known memory leak.
Lastly, I also write 6502 assembly for fun on occasion, so I'm probably not the best benchmark for what's reasonable here ;)
Oh, that would be really cool. Wasm does already look quite similar to core forms, particularly now so with the stack switching proposal. I would love to see a syntax-case expander atop Wasm...
Have you heard of Hoot? It's more than just guile, it has a whole toolkit for WASM.
I did check it out to see how the pros do it but it was a whole lot of code and I ended up implementing the core ‘Lisp’ in plain WAT (≈1 000 lines but can probably be reduced.) I would like for the whole toolkit to fit in my smol head.
I worked on a similar idea with Gwe, the idea being that to generate efficient WebAssembly for Derw (ML-family), it would be a lot easier for me to manually optimize if the language was slightly more high level. I lost motivation at some point along the way, so it's half finished as an idea. But maybe one day I'll get that craving again
My hunch is there is a lot of dependence on always-on internet servers that exists primarily for syncing data privately between a small number of devices - in particular our own. If you bolted on a good CRDT sync API to an existing ecosystem like the syncthing daemon, such that lots of applications could hook in and work with structured data with their own schema instead of files on disk (collisions!), many things could become private P2P. Contacts, calendars, RSS state, todos, notes, …
I am actively working with this, but I think it's an idea that deserves a wider spread of attention: Building programming languages with a Dictionary<string, Stack> (I've called it a Multistack, recently a friend suggested that Tray might be a better name) at their core results in something -surprisingly- expressive.
Like dynamic scope… what languages still have it, other than elisp? Emacs makes extensive use of dynamic binding to make temporary configuration changes. https://www.gnu.org/software/emacs/manual/html_node/elisp/Dynamic-Binding-Tips.html
set / show rules in Typst are dynamically scoped! Ordinary bindings are lexically scoped, however.
It seems to be a good use case in Emacs and it is definitely a good use case in Typst. I am similarly curious as to what other languages have dynamic scope and use dynamic scope well...
It is like dynamic scope, but in my (limited) experience, trays are handier because you don't have to push/pop entire scope frames. They also fit better with concat languages
Yes! One random little example: you want to pass a Log around for logging. If you do that on the main stack, it gets in the way. But if there's another stack named log, you pass it there and now it's not in the way. Why does log need to be a stack instead of a single value? So that you can e.g. use different loggers in different parts of the code. To use a different logger during a function call, you push the logger, call the function, pop the logger.
I imagine there are use cases like that but where you want an auxiliary stack with elements of different types, too, though I don't have a worked example off the top of my head.
My favorite trick is using a stack for what OO languages use self/this for. Plays really nice with concat languages
Do you have an example of how this is expressive? Sounds very interesting but can't think of anything else than URL query params at the moment
https://nova-lang.net, https://github.com/mirth-lang/mirth and https://stk.junglcoder.com are all examples of working with it in one way or another.
You might want to take a look at Kernel (https://web.cs.wpi.edu/~jshutt/kernel.html) and how it carries around environments as first class objects in order to manage lexical and dynamic scope.
A nix-style binary caching system + torrents could be used for building and distributing shaders for games.
UI libraries for the web are awful. They either lack important components, are not accessible, are over complex or multiple of these. I think a web components based approach with some prebuild themes could work but that's a lot of work.
I think most people would agree that web ui frameworks are in a very sorry state right now. Web components are a fundamentally bad building block for a ui framework though. They lack composability (one component can only pass data to another as string attributes) and, as long as you’re making a framework, they don’t really get you anything.
Web components are great for delivering a single, framework agnostic, reusable component and that’s about it.
framework agnostic, reusable component
Well, that's what I want. I'd probably do the rest in CSS & Rust and use the web components only for the required interactivity.
The major reason for lack of adoption of academic programming languages is unfamiliar syntax.
I would love to see a purely-functional language that makes a commitment to imperative syntax. The use of do-notation in languages like Lean get close to this, but not close enough -- I would like to see a language with imperative syntax like Python or Java that desugars to or is defined in terms of well-structured primitives, that track effects / state throughout the codebase and expose this to the user when they want it.
I particularly think effect handlers would be a good basis for this. I would love to see a language fully centred around them, that hides this interface from the user and instead exposes modules with syntax/semantics like throw, yield, etc. The user can inspect their implementations and understand the unfamiliar higher-level primitives underlying them if they so want, but need not do so to successfully use the language -- the exposed abstractions should be well-understood broadly.
What do you think of this idea, where syntaxes in the same equivalence classes, roughly speaking, could be used interchangably and even be configured like we could do with coding styles or editor themes today? Bridging semantic differences is another matter entirely (like legal systems across different countries): control transfers between regions of code needing special handling at interfaces/boundaries (''borders''), and handling of ownership transfer and references to runtime objects and their semantics (''citizenship'' and ''diplomacy'' issues), just imagine how borrow checking is supposed to work with subtly different semantics in different regions, as well with regions without borrow checking at all.. Composability of semantics(es) is going to be difficult, I wonder what an ''European Union''-style approach (lol!) to programming semantics, in an analogy to legal systems, would turn out to be like..
I don't like it, but you definitely should check out Unison. It's one of the only (textual) programming languages I'm aware of where the representation is not textual. And, as someone else mentioned, Racket's hash-langs serve a similar function. I just only don't like it because for me, the point of having imperative syntax would be learnability, through analogy with the popular imperative paradigm -- and I think inconsistent syntax will hinder learnability, rather than help it. No other reason!
I would say with composability of semantics: extraordinarily hard problem, yeah. I think it's gotta be via FFI: I don't think extensible semantics works for compositionality, because if you're extending the semantics, you don't want to just add semantic forms, you also want to modify and remove existing semantic forms, and that breaks compositionality without a mapping back -- which you likely won't have, because if your extended semantics are mappable to the starting semantics, you wouldn't bother extending your semantics in the first place. (This is a little hard to talk about precisely. I'm being vague here. You might like reading my friend's blog post on semantic extensibility, though.)
Anyway, even FFIs are an understudied area of research. The two most interesting things here are Wasm and crABI (extending the C FFI with rich type info), IMO, but they only even provide an interface, and leave compositionality / a lack thereof up to the implementator...
a purely-functional language that makes a commitment to imperative syntax.
They're not completely functional, but OCaml and especially F# fit this idea for me. They're functional languages salted and seasoned enough for an imperative programmer like myself. I normally write C++ but I've used a bunch of other languages and F# doesn't feel that foreign.
that track effects / state throughout the codebase
This sounds like a great concept around which to write an embedded and sandboxed scripting language (like Lua) for other languages (C/C++/Rust).
This sounds like a great concept around which to write an embedded and sandboxed scripting language (like Lua) for other languages (C/C++/Rust).
Essentially, this is already done through WebAssembly. The fast stack switching proposal is exactly effect handlers, just from an operational perspective. And they already have interface stuff. Between the two of those, you can handle language-internal and language-external effects pretty handily.
I'm trying part of this within Haskell, aiming to make a synthesis between functional and imperative. Since it's in Haskell it only goes one step of the way. There will generally be a lot of other unfamiliar syntax floating around. Anyway, I'm doing it my effect system Bluefin, and trying to encourage writing in do notation. I find it makes for much more readable programs.
Good luck with Bluefin! I would say, I'm normally skeptical of the dependency-injection tradition of effect handlers (I learned them in the more operational "jump around the stack" style), but Haskell is probably the language in which you have enough trust in / control over the compiler to make it work.
I have been hitting performance issues in my Scheme library, even. Since Scheme has delimited continuations + macros (untyped) effect handlers are pretty easy... the Guile Scheme call-with-prompt and abort-to-prompt are exactly shallow effect handlers, even. But you don't have enough / any type information to infer / enforce linearity of resumption and so you can't do the fast stack switching optimization, which tanks performance... so, despite wanting such a language to be as simple and normal as possible to attract non-programming languages nerds, I think it should have some form of ownership just to enable that stack switching.
(Maybe dynamic enforcement, like WebAssembly, is good enough though.)
Thanks! Using Haskell type system to ensure scoped use of resources (what I call the "analytic" approach to effect systems) has proved practical for my purposes. For those interested in stack-based effects there is a similar API over GHC's delimited continuation primops: https://hackage.haskell.org/package/bluefin-algae (that's not by me, but by Li-yao Xia). I also have some interesting ideas about how to use GHC's LinearTypes extension useful for ensuring linearity-like properties, as you refer to.
Good luck with your project! In general I really like your idea of embracing imperative syntax for functional programming. I think it will make functional programming much more approachable without having to sacrifice its valuable qualities. I can get some of the way in Haskell with do notation, but there will still be other parts where a codebase is not using "imperative style" and a lot of scary syntax floats around.
Have you tried ReScript?
EDIT: I just finished reading your comment about the state/effect tracking, and actually, now I don't think ReScript solves that problem. But it does provide a nice "imperative" syntax over what's essentially OCaml under the hood.
I'd love to chat about this. I'm currently working on an effect-based language (like so many are) and while I'm in the middle of thinking out the core calculi, I'm very sympathetic to the idea of having the language present as something very familiar on first blush.
When building a compiler, get the codegen part right /first/, then think about the front-end (parsing) and core.
This is not my own idea (I think back-to-front compiler design was originally coined by Lindsey Kuper) , but I recently relearned this lesson the hard way with a thorough self-own (have to throw away a codegen stage because of poor assumptions made in the front).
For Inko I started with a simple S-expression based syntax (using some existing library) and focused almost exclusively on the (at the time) interpreter and general language semantics. I recommend taking a similar approach (though perhaps by using something other than S-expressions), as far too many people get hung up on irrelevant syntax details/discussions rather than focus on making an actually working compiler.
I think you could do pretty well by taking a Linux kernel and writing a new userland atop it entirely in the Erlang VM. Basically just use the kernel for its device drivers.
Synit is trying something like this, but with their own Syndiated Actors rather than Erlang actors
Ha, thanks to your reminder I just found out that synit's author is now working in Maastricht University (since 2025 maybe?), excellent news! Go public sector.
They did something similar for Go, where PID 1 is just a Go process that spawns whatever goroutines it needs and has a zombie reaper.
This is basically what Nerves is:
https://github.com/nerves-project/nerves
Nerves itself is somewhat clunky as a general-purpose userland as its intended use case is single-application deployments to small hardware. They do have some cool components including a GUI system, though it assumes a fixed client resolution.
I'm aware! Nerves would definitely be my starting point, just for the build infrastructure. <3
I have a hunch that more and more of the linux kernel will move into userspace via mechanisms like io_uring or others. In doing so we can have much more customized and much more diverse systems.
My hunch is Terraform is not the best we can do by a long shot.
The most successful Terraform competitors repackage Terraform's providers and implement the same state/plan/apply core.
There might be better ideas out there, but we haven't seen them in ten years.
I was going to point you to System Initiative but it seems two distinctly bad things happened : they went AI and then archived the repo https://github.com/systeminit/si#system-initiative
I never got it to run in order to know if it was as much of the second coming as Adam claimed but it is my recollection that it was trying to innovate
They keep pushing "Swamp" but its website is so obscure I can't figure out what it is supposed to be doing. I guess that's par for the course for AI startups these days.
as best I can tell from https://github.com/systeminit/swamp#quick-start and the massive number of "steering documents" that it emits on swamp repo init I'd guess it's very similar to the problem that https://github.com/github/spec-kit is trying to solve: generate enough .md files that somehow cause the floating point matrices to stay on task. I believe that's the path that AWS's Kiro and Google's Antigravity are trying use, too
The README does speak about workflow execution, but based on the rest of the repo, I have every confidence it's a tool that uses agents to read markdown to generate more markdown for more agents to read to end up doing a thing. I'm sure there's a fascinating reason why having it just generate ansible yaml is inferior, aside from the VC money and similar "moar AI"
My observation for many years has been that inlining subresources in HTML is ridiculously good for performance.
The theory is, keep subresources separate and then they can be cached and loaded immediately rather than redownloading.
The practice is, for many sites, subresources are not in the cache a lot of the time; and that even when they are, fetching from the cache can easily be slower than having it inline. The caches can be astonishingly bad—Chromium in some situations races network fetch and cache load! But even at their best, I think they’re worse than people expect.
There are also optimisation opportunities, because you can remove unused CSS and JS, bringing sizes and runtime cost down even further. And compression algorithms generally favour compressing one large blob rather than two small blobs.
It depends on the nature of the subresources, latency and bandwidth; but my feeling is, and this is particularly where I haven’t investigated it properly: the threshold where inlining mandatory subresources is almost always equivalent or better in performance is far higher than most people imagine. Like, probably more than 100 kB compressed. Which is more than any content site should ever need of CSS and JS.
My new website (maybe deploying today?) will default to inlining all CSS and JS, though there’ll be a “dev mode” checkbox/cookie to get it to serve commented, unminified, possibly-less-inlined pages because that sounds fun. Fonts (if used, not by default) and any images will be the only external subresources, for now.
Gatsby has realized this and inlined all CSS into every subpage for a while. Unfortunately the size of the output is now massive and for larger static sites it can lead to issues during deployment. In practice you could accomplish the same thing with preload headers and without bloating storage, but no static site host does this.
What if you removed all the overhead of tree data structures and objects from web browsers and borrowed the ECS idea from games instead? When I think about it seems to fit the problem very well. The problem is that to evaluate the idea you have to build a browser engine, which is a pretty big hurdle.
One I will surely never get to test: there is a meaningful space for a language that carries over a lot from Rust, such as lifetimes, memory and race safety, and a rich static type system, but changes other tradeoffs to optimize for applications over systems programming, e.g. perhaps not making all the same guarantees about whether things will live on the heap or stack, and offering simple-but-slow 'outs' like garbage collection and (this is the handwaviest part!) some simple-but-slow concurrency model you can always fall back to--message passing? a special shared heap or shared objects with coarse-grained but safe locking? (The other approach concurrency is what Java does, where you're not well-protected from strange program behavior due to races, but races don't break type safety like they do in some languages.)
Anyhow, I think you could meaningfully improve on the performance of other application languages; for example, you could hopefully write libraries or "leaf" code in such a way that it's not going to produce garbage, etc., and 'real' monomorphized generics and so on help you write powerful libraries. You could also meaningfully improve on most other application languages' safety. The hope is that while Rust is a local maximum and great at the target use case, it isn't the only good point in the design space that uses some of its main ideas. Finding another good point is a rather large exercise for the reader though!
If terminals behaved like REPLs instead of teletypes, it would unlock a lot of power, not just for devs, but for ordinary users. (Obviously not for full-screen terminal apps, but the 99% use case of enter a command, get some output.)
Benefits:
Oils has done some work on this with their FANOS protocol. (Hi Andy!).
I use Warp (from before its AI phase) specifically because they were one of the few terminal programs actively attempting something like this. For the most part, it's been super-nice.
I'm not 100% clear on what would truly be unlocked by making terminals REPLs, but my hunch is it's a lot more than we expect.
Code location of unit tests matters when it comes to quality of LLM generated code. I.e. languages where the tests reside in the same file as the implementation are easier for an LLM to generate well as those where they can live in a completely separate package.
markov chains over sequences of hand-engineered features extracted from the ASTs of the training corpus would perform just as well as an LLM for generating code, especially if using beam search to optimize the total probability of the whole generated source file instead of generating one token at a time.
I think the last 10-15 years have provided a lot of evidence toward what an ideal modern systems programming language should look like. The emergence of Rust, Zig, and modern C++ has allowed much of that tradeoff space to be properly explored empirically.
None of Rust, Zig, or C++ are "it" but most of the critical elements are reflected across at least one of these languages. I think it would be possible to synthesize a really nice systems programming language from scratch that ties most of these learnings together.
As someone unfamiliar with Zig and modern C++, I'd be interested to hear more as to what you're thinking! The major advantages of Zig and C++ over Rust that I'm aware of are compatibility with C (zig cc and clang) and better support for compile-time code (Rust const is improving, though). Curious if there's anything else you had in mind.
The compile-time metaprogramming, variadics, and reflection is a critical part. You can solve a lot of problems with it if the capability is strong enough. Zig compile-time is tidier than C++. Some additional things off the top of my head:
The ability to easily constrain types at the point of definition with both compile-time and runtime checks without writing asserts e.g. no one should be using raw integers as function arguments. You can do this in C++ but it isn't a first-class feature. A native language facility would aid formal verification.
Native bit-structured types. Think bit fields but with the properties and composability of normal byte-structured classes. You can get about 80% of the way there in C++ with some hackery but it isn't fully transparent and some things that could be possible in theory don't work.
The ability to reason about memory with out-of-process ownership i.e. it isn't known at compile-time. This is a common case for high-performance I/O. Rust isn't really designed for this and even C++ couldn't properly handle some edge cases until C++17.
Optional type ACLs on functions and variables. This is a generalization of the passkey idiom: you can annotate any function such that only specified types can access it. It is much more granular than public versus private, documents code dependencies, and helps keep Hyrum's Law at bay. C++26 added variadic friends to expand usage of this pattern.
A stackless continuation passing style language similar syntax to haskell's IO Monad and enforced tail call optimization (there is no stack so nithing else is even possible), could be more efficient than traditional languages. Though I realize you eventually end up reinvenrting the stack as a set of closures and passed continuations...
rebol was on to something seriously good, not so much for the language as for the runtime. I think other languages could stand to learn from it.
There is an algebra of programming that is based on Generic Programming concepts (Stepanov) that allows the same proof and derivation niceties as Bird's algebra. Bird's algebra relies too much on the inductive structure of lists and the the optimizations available in GHC to derive the efficient versions of the algorithms.
I think there's probably an interesting way of fusing together databases and programming languages (not sure whether this is a programming language with an embedded db/better persistent programming or a database with a good embedded programming language -- probably more of the latter). The backend/DB split has never sat well with me since processing should happen where the data lives, but plsql is kind of a mess to work with. Also, execing giant SQL strings for bulk inserts and constructing queries with string concatenation seem like giant red flags that the abstraction sits in the wrong place. The relational model is great, I just want an imperative layer that lets me drop down into the guts of the DB side more easily.
There are a few existing things that might or might not approach what you are thinking of:
Embedded SQL, the ISO standard way for interfacing programming languages such as C to a database, eg https://www.postgresql.org/docs/current/ecpg-concept.html
LINQ to SQL https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql/linq/
Writing stored procedures etc. in languages other than Ada-ish extended SQL https://wiki.postgresql.org/wiki/PL_Matrix
WRT bulk data, Pg has COPY and DuckDB has even better answers.
There’s potential for a synthesis of something like DuckDB’s data access layers, programmed using an imperative language with LINQ-ish features, with the possibility of pushing expressions down into the query plan.
I like the idea of avoiding runtime bounds checks with size-dependent types, but I haven't explored whether that is feasible in terms of compile times and developer ergonomics in a language used by industry.
I might be wrong, but I think Ada might already have this.
As well as Ada there’s ATS. I think a modern approach to range types would involve control-flow aware value range inference, so that range type refinement understands the arithmetic and comparison operations performed by the code. I expect this would make overflow-safety far more obvious; whether it can avoid being far more obnoxious is another question…
Why can't a process handle system calls for a child process? It doesn't have to, but it would seem such a facility would make it easier to create sandboxes and even experiment with new system calls. The idea came to me when when I wanted to run an MS-DOS 1x executable on Linux (using the vm86() system call) and I had to handle INT 21h calls myself.
You can via ptrace's PTRACE_SYSCALL. Alternatively you can run the child under kvm and service syscalls with your own kernel, or turn them into vmexits that your host turns into syscalls again - tinykvm does this.
It turns out this isn't actually very useful for sandboxing, because there is a lot of implicit resources that can be addressed by syscalls that aren't immediately obvious as being operated on from syscall arguments (/proc/pid/fd, openat with relative paths or directory fds, etc). In a good sandbox you really want to control access to transitive resources as well. Chomium's libsandbox does something kind of similar, by installing a seccomp filter that restricts syscalls that shouldn't be used, but it also needs to turn some syscalls into signals that it handles in the same process and turns into IPC calls to the broker process for some things. It also installs a lot more capabilitity-level restrictions via Linux namespaces that the kernel will use when processing syscalls that cover the vast majority of complex transitive behavior, however.
At some point in the future, it will become common for sighted people to ask voice assistants to read webpages out loud to them while their eyes are otherwise occupied, e.g. driving or walking.
The vast majority of modern software could run significantly faster on the same hardware, given suitable efforts.
If declarative system/infrastructure description was good so would just having a DB schema be. Instead we have DB migrations, so maybe infrastructure should and systems should also be managed as a set of migrations. This would also make fixing things so much nicer instead of always being a bit special. Declarative description of a target state always comes with its abstraction shining through, so I think even though it feels nice, just like just having only a DB schema feels nice, something like a migration path could be a better approach to get there and especially to maintain it long term.
I think if you properly isolate all nondeterminism to some kind of polymorphic effect system, you could build insanely good debugging and testing UX simply by recording your inputs so they can be played back later. I want interactive querying of any detail of a particular failing execution! I want a crash dump to give me a full reproduction case! I want something like Glamorous Toolkit that gives me total observability into a running program! And that all becomes extremely tractable once you have the ability to just rerun the code without side effects.
Less a hunch and more rant, but I still don't know why backups and archiving have to be so annoying and terrible.
Not a mac user (except work macbook where my backup fits on an encrypted thumb drive) so maybe time machine + keep buying disks is a lot easier, but this household uses:
and it feels every single use case needs a different strategy and tech and none of them are easy or frictionless (or cheap).
Large compressed JSON (or XML etc) documents could be parsed much more efficiently with a parser that cut across both layers, such that common strings (or even entire subtrees, though this is probably harder) that are represented as backreferences in the input can remain as single copies in memory.
The tricky bit will be that the backreferences are probably more like ,"foo": than "foo" - but if you store the contents of each backreference as a token stream, and have some sort of context tracking so you can reparse a backreference if it suddenly appears in the middle of a string or something, I think you can make it work.
I know that html is not a context free grammar and that replicating websites from archive.org is difficult if you just make a bunch of regex's to parse the urls and get the resources. It's easier to use playright to scrape a website, monitor the network resources and download them especially for complex sites that use javascript to dynamically load resources. But I want to write this out formally and then reprove it to myself by building out projects in a scientifically controlled way.
Also, I want to deep dive into how duckdb maps queries to parquet files in bytes. I want to understand if there's ways for it to pull exactly the bytes it needs from a parquet file instead of blocks of data.
A graphical programming language, competently designed and targeting skilled programmers, would be easier to read, easier to manage in source control, have fewer bugs, be easier to use from agents, and overall just be a better experience from dev to machine.
Runner-up: 100% of the existing surge in Terminal support is because we flubbed remote access, but that hunch is going to be at least in part a social solution, so I don't think it'd strictly qualify.