Apple's new desktop computers are designed specifically for local AI development
3 points by snej
3 points by snej
The devices’ popularity for production inference took off after macOS 26.2 shipped last December. According to Apple’s release notes, 26.2 enabled “low-latency communication between Thunderbolt 5 hosts for use cases including distributed AI inference using MLX.” Thunderbolt 5 is a very fast wired data connection, and MLX is an open source array framework designed to help machine learning workflows take full advantage of the M-series chips’ unified memory.
Since then, both hobbyists and professional developers and researchers have been essentially daisy-chaining Mac minis or Mac Studios to run inference on local large language models that are much bigger than anything that could run a single mass-market device—providing an alternative to ultra-beefy specialized hardware featuring specialized Nvidia GPUs.
There was some way to direct connect 4x of the studios (using Thunderbolt) with 512GB each into a 2048GB cluster running an open model across the four nodes. Been a while since I read the blog article on it.
At an eye-watering USD$20,819 a pop I guess it actually is still cheaper than the $275,300 for 8x 80GB H100s and probably ironically costs less in electricity to run them