ty: An extremely fast Python type checker and language server
119 points by jparise
119 points by jparise
I flagged this as "already posted" because two weeks ago ty was a Python type checker by Astral, and today... ty is still a Python type checker by Astral. The only thing that's changed is that it's now at beta status, but that's basically just business news about a product cycle, which ought to be off-topic for lobste.rs. If and when a stable public release is out, I think that'd be a good candidate for a post, but even that's probably going to require going more than two weeks between posts advertising this product.
it's a release milestone announcement. Release announcements are on lobsters all the time. And these are different pages. The other one is just someone linking the project docs site, this is an actual announcement, dated the day it was posted, with entirely different content.
Astral being "ready to recommend it to motivated users for production use" is a big deal for my team to be able to use this at work, so I definitely appreciated this being posted here.
This beta announcement that they're recommending it to "motivated users for production use" is relevant to me and I'm happy it was posted here, since this is where I saw it :) Not that I ncessarily disagree with the "already posted" flag.
business news about a product cycle, which ought to be off-topic for lobste.rs
Should it? https://lobste.rs/search?q=%22announcing+rust%22&what=stories&order=newest
Personally I feel the non-profit Rust Foundation and the VC-backed Astral company are quite different entities.
Wow, it's hard not to come across as confrontational on the internet! I don't wish to throw shade on either your clearly shared sentiment or for-profit organisations striving to do good in the ecosystem. I would totally understand if OP comes across as hypocritical. I just wanted to share another point of view.
Recently in the (public) mod log:
The latter in particular seems very on point here.
I think that if this were a link to a changelog for a new version then it would be more comparable to the results in the search.
I like ty etc but yeah feels like this was discussed 2 weeks back.
(Aside: would love to see a post going deep onto salsa, the lib that ty uses for caching of looooots of results. Had a chance to poke at ty's source code and all that was real fun to think about)
I saw "extremely fast Python..." and thought "this must be by the folks who make uv and ruff". They simply do not miss.
Is the ty vs. Pyrefly comparison really apples-to-apples? Aren't those two have their own, different, type systems?
Or does Python have its own type system now that all of these tools implement?
Yes, Python specifies a type system using PEPs and implementors conform to it (in theory, anyway) https://peps.python.org/topic/typing/
Very excited for increased stabilization. mypy checks are somewhat of a bottleneck in our Python CI, while ty (not yet adopted) runs the "same" checks essentially instantly, even for our larger subprojects.
The most striking thing about this to me is how their benchmarks blow Pyrefly out of the water. And Pyrefly is also written in Rust and also has benchmarks where they show they're faster than Pyright and Mypy: https://pyrefly.org/
Ironically, Pyrefly is a reimplementation of Facebook's Pyre typechecker from OCaml to Rust (also, they claim, for performance reasons).
One wonders if Pyrefly has a raison d'être any more.
Note that the biggest performance wins that we quote are for incremental computation — e.g. during live editing as part of a language server, or when running ty in the background with ty check --watch. We are using salsa as our incremental computation library, which lets us add incrementality at a very fine-grained level. (Most of the time down to the level of an individual definition or scope, but in some cases down further to the level of individual expressions.) When we reprocess your project in response to some edits, that greatly reduces the amount that we have to recompute.
My understanding is that Pyrefly's incrementality is at the module level, which explains why it has to recompute so much more. Though I believe they are also investigating making their incrementality more fine-grained like ty's.
We are using salsa as our incremental computation library, which lets us add incrementality at a very fine-grained level. (Most of the time down to the level of an individual definition or scope, but in some cases down further to the level of individual expressions.)
Are you adding each expression as a tracked entity to the Salsa database?
How are you finding the Salsa entities to update when the source code (text) changes? You'd need to diff before vs. after, map the changed AST nodes to Salsa entities to update. It'd be interesting to look at the code that does this.
Are you adding each expression as a tracked entity to the Salsa database?
Not every expression. But there certain expressions that benefit from being tracked, and for those we do add them as salsa entities. (Search for "standalone expression" in the ty_python_semantic crate)
How are you finding the Salsa entities to update when the source code (text) changes? You'd need to diff before vs. after, map the changed AST nodes to Salsa entities to update. It'd be interesting to look at the code that does this.
Here we're leveraging that Salsa will notice if the output of a tracked function is the same as before, even if the inputs have changed, and short-circuit recalculating anything further downstream. Concretely, that means that we do need to fully re-parse the file, and all of our tracked data that mentions e.g. source locations will become invalidated and recomputed. But once we get down into type checking proper, we quickly notice that the types we have inferred for the unchanged parts of the file are the same, and then don't have to recompute anything derived from those.
I think Meta prefers maintaining its own tooling, if only because they can tailor it to their needs without external influence.
React is the big exception, but I feel like FB's language tooling endeavors seem to not gain much traction.
a very long time ago, before LSP, before treesitter (and therefore before ast-grep), they had this thing called pfff, which was for doing static analysis and source-to-source transformation for code refactoring. It was really powerful, and at the time, one of the best ways to do that sort of sophisticated large scale mutation on PHP codebases. While at Etsy I worked on a migration in Etsy's very large PHP monolith and we used this and it was extremely useful. It has always shocked me how powerful this tool was and how few people have ever heard of it. Truly ahead of its time stuff.
this does remind me that libcst is from instagram, also a very useful lib.
I respect the work they do, just was commenting more on the communtiy stuff
Also in the JS ecosystem:
yarn did okay for a while but is now I think very much a minority.watchman I think is still widely used because it's used by jest.jest I think is still popular? It was recommended by default by React for a while.flow typechecker. The community around that died very quickly because they broke backwards compatibility so many times. Shame because it had some nice features like being faster than tsc for a while.Jest was popular for a bit but I think Vitest is overtaking it. As for Flow, its biggest user is probably React.
Ah yeah. React Native uses Flow too last I checked. It's a slightly funny situation because React and RN are both developed using Flow but the overwhelming majority of users use them with TypeScript types instead (if any).
vitest overtaking jest is pleasing but not surprising because jest is IME slow as the hills.
SemVer-compatible upgrade enforcement
Does anyone know anything more about this? Would be interesting if they will also use Trustfall like Cargo semverchecks does. The maintainer said that he actually build a prototype of that.
I can't seem to find the actual Rust implementation in the ty repo or in their GitHub org. The blog post says that ty was built in the open and has an MIT license. Does anyone know where the actual code ist?
It’s in the repo for ruff, their Python linter:
The confusion is definitely unfortunate. To clarify, the reason for this is that ruff and ty share a lot of common code under the covers. When we first started working on ty, it was most expedient to develop it all in the same repo. We also envision that ruff will grow some type-aware lints at some point down the road, at which point ruff will take on a dependency on the crates that implement the type checking logic.
Following the Beta release, our immediate priority is supporting early adopters. From there, we're working towards a Stable release next year, with the gap between the Beta and Stable milestones largely focusing on: (1) stability and bug fixes, (2) completing the long tail of features in the Python typing specification, and (3) first-class support for popular third-party libraries like Pydantic and Django.
I'm glad to hear them mentioning Django support is on their roadmap. I'm not entirely sure how it will shake out but their execution on everything they attempt is damned good.