Decentralized Training Can Help Solve AI’s Energy Woes
Artificial intelligence harbors an enormousenergy appetite. Such constant cravings are evident in thehefty carbon footprint of thedata centers behind the AI boom and the steady increase over time ofcarbon emissions from training frontierAI models. No wonder big tech companies are warming up tonuclear energy, envisioning a future fueled by reliable, carbon-free sources. But whilenuclear-powered data centers might still be years away, some in the research and industry spheres are taking action right now to curb AI’s growing energy demands. They’re tackling training as one of the most energy-intensive phases in a model’s life cycle, focusing their efforts on decentralization. Decentralization allocates model training across a network of independent nodes rather than relying on one platform or provider. It allows compute to go where the energy is—be it a dormant server sitting in a research lab or a computer in asolar-powered home. Instead of constructing more data centers that requireelectric grids to scale up their infrastructure and capacity, decentralization harnesses energy from existing sources, avoiding adding more power into the mix. Hardware in harmony Training AI models is a huge data center sport, synchronized across clusters of closely connectedGPUs. But ashardware improvements struggle to keep up with the swift rise in size oflarge language models, even massive single data centers are no longer cutting it. Tech firms are turning to the pooled power of multiple data centers—no matter their location.Nvidia, for instance, launched theSpectrum-XGS Ethernet for scale-across networking, which “can deliver the performance needed for large-scale single job AI training and inference across geographically separated data centers.” Similarly,Cisco introduced its8223 router designed to “connect geographically dispersed AI clusters.” Other companies are harvesting idle compute inservers, sparking the emergence of aGPU-as-a-Service business model. TakeAkash Network, a peer-to-peercloud computing marketplace that bills itself as the “Airbnb for data centers.” Those with unused or underused GPUs in offices and smaller data centers register as providers, while those in need of computing power are considered as tenants who can choose among providers and rent their GPUs. “If you look at [AI] training today, it’s very dependent on the latest and greatest GPUs,” says Akash cofounder and CEOGreg Osuri. “The world is transitioning, fortunately, from only relying on large, high-density GPUs to now considering smaller GPUs.” Software in sync In addition to orchestrating thehardware, decentralized AI training also requires algorithmic changes on thesoftware side. This is wherefederated learning, a form of distributedmachine learning, comes in. It starts with an initial version of a global AI model housed in a trusted entity such as a central server. The server distributes the model to participating organizations, which train it locally on their data and share only the model weights with the trusted entity, explainsLalana Kagal, a principal research scientist atMIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) who leads theDecentralized Information Group. The trusted entity then aggregates the weights, often by averaging them, integrates them into the global model, and sends the updated model back to the participants. This collaborative training cycle repeats until the model is considered fully trained. But there are drawbacks to distributing both data and computation. The constant back and forth exchanges of model weights, for instance, result in high communication costs. Fault tolerance is another issue. “A big thing about AI is that every training step is not fault-tolerant,” Osuri says. “That means if one node goes down, you have to restore the whole batch again.” To overcome these hurdles, researchers atGoogle DeepMind developedDiLoCo, a distributed low-communication optimizationalgorithm. DiLoCo forms whatGoogle DeepMind research scientistArthur Douillard calls “islands of compute,” where each island consists of a group ofchips. Every island holds a different chip type, but chips within an island must be of the same type. Islands are decoupled from each other, and synchronizing knowledge between them happens once in a while. This decoupling means islands can perform training steps independently without communicating as often, and chips can fail without having to interrupt the remaining healthy chips. However, the team’s experiments found diminishing performance after eight islands. An improved version dubbedStreaming DiLoCo further reduces the bandwidth requirement by synchronizing knowledge “in a streaming fashion across several steps and without stopping for communicating,” says Douillard. The mechanism is akin to watching a video even if it hasn’t been fully downloaded yet. “In Streaming DiLoCo, as you do computational work, the knowledge is being synchronized gradually in the background,” he adds. AI development platformPrime Intellect implemented a variant of the DiLoCo algorithm as a vital component of its 10-billion-parameterINTELLECT-1 model trained across five countries spanning three continents. Upping the ante,0G Labs, makers of a decentralized AIoperating system,adapted DiLoCo to train a 107-billion-parameter foundation model under a network of segregated clusters with limited bandwidth. Meanwhile, popularopen-sourcedeep learning frameworkPyTorch included DiLoCo in itsrepository of fault tolerance techniques. “A lot of engineering has been done by the community to take our DiLoCo paper and integrate it in a system learning over consumer-grade internet,” Douillard says. “I’m very excited to see my research being useful.” A more energy-efficient way to train AI With hardware and software enhancements in place, decentralized AI training is primed to help solve AI’s energy problem. This approach offers the option of training models “in a cheaper, more resource-efficient, more energy-efficient way,” says MIT CSAIL’s Kagal. And while Douillard admits that “training methods like DiLoCo are arguably more complex, they provide an interesting tradeoff of system efficiency.” For instance, you can now use data centers across far apart locations without needing to build ultrafast bandwidth in between. Douillard adds that fault tolerance is baked in because “the blast radius of a chip failing is limited to its island of compute.” Even better, companies can take advantage of existing underutilized processing capacity rather than continuously building new energy-hungry data centers. Betting big on such an opportunity, Akash created itsStarcluster program. One of the program’s aims involves tapping into solar-powered homes and employing the desktops and laptops within them to train AI models. “We want to convert your home into a fully functional data center,” Osuri says. Osuri acknowledges that participating in Starcluster will not be trivial. Beyond solar panels and devices equipped with consumer-grade GPUs, participants would also need to invest inbatteries for backup power and redundant internet to prevent downtime. The Starcluster program is figuring out ways to package all these aspects together and make it easier for homeowners, including collaborating with industry partners to subsidize battery costs. Backend work is already underway to enablehomes to participate as providers in the Akash Network, and the team hopes to reach its target by 2027. The Starcluster program also envisions expanding into other solar-powered locations, such as schools and local community sites. Decentralized AI training holds much promise to steer AI toward a more environmentally sustainable future. For Osuri, such potential lies in moving AI “to where the energy is instead of moving the energy to where AI is.”
No discussion yet. Be the first to share your thoughts!