AfirkaLLM Part 3: No single machine could do this

How a university lab trains a model on borrowed compute, and why every dataset we touch stays on its owner’s terms.

This is part 3 of a four-part series: 1. Where the data comes from, 2. What the experiment found, 3. Compute and terms, 4. What is missing and how to help. Part 4 will be published over the next few days.

Running the five 5-billion-token pre-training experiments from Part 2, and soon one much larger full run, takes a lot of GPUs. Here is what the headline numbers hide: no single compute source could provide them. Getting this done meant combining a portfolio of compute partners, each used for what it does best. Because every run used exactly the same recipe, with the same model, the same settings, and the same 5 billion tokens split into the same 9,536 training steps, the sweep also became a fair side-by-side test of two kinds of hardware.

• Google Cloud (Vertex AI), with Gemma credits, gave us training at scale without running our own machines, and the base model itself. Three of the five ablations ran here, and they ran like clockwork: each held about 16,700 tokens a second, kept its GPUs busy 99% of the time, and finished in about 84 hours with less than an hour and a half lost along the way. But even generous cloud credits hit a hard wall: our cloud region could not give us enough large-memory GPU machines at the same time to run the whole sweep at once. Capacity, not budget, was the bottleneck. That is the constraint that most surprises people who assume “cloud” means “unlimited”.
• Cassava’s H200 cluster filled exactly that gap. Its GPUs are a newer generation with far more memory (141 GB against 80 GB), and, critically, they were available when the cloud region was full. The two ablations Google Cloud could not fit, baseline and math, ran here and unblocked the entire experiment. On identical work, the H200s processed about 22,300 tokens a second against 16,700 on the A100s, a 1.34-times speed-up, and needed about 62 hours of training instead of 83. That speed comes straight from the newer chip: more computing power and much faster movement of data in and out of its memory. It is also the natural home for the large full run to come. That African-language AI is being trained on African compute infrastructure is not incidental. It is the kind of capacity the continent needs to own its own models.
• Local hardware, an NVIDIA DGX Spark from a 2025 academic grant, plus A6000 and RTX 4090 machines, carries the quick, everyday work: building the pipeline, training the language ID and topic classifiers, running evaluation, and hosting the review platform. This is fast iteration, off the cloud. Keeping that work local means paid credits go only to runs we have already tested locally.

Figure 1. Faster hardware, shared with other users. Hours each ablation spent training on its 5 billion tokens, and time paused between stretches of training, by where it ran. The math run was paused and resumed partway, so its paused time covers only the stretch after it resumed. Source: AfirkaLLM training logs.

Speed on paper is not speed in practice. The comparison is not entirely one-sided. The A100s ran at 99% of capacity and were almost perfectly steady, losing under an hour and a half across a roughly 84-hour run. The H200s were kept busy only 82 to 92% of the time, and their runs had more breaks. That was by design: Cassava’s cluster is shared, so at times we paused our runs to let our collaborators use it, then picked up from our saved progress once they were done. The baseline run lost about 14 hours to these pauses, and the math run was paused and resumed partway. Part of the lower figure is those pauses; the rest the logs do not explain, though the GPUs were likely waiting for data to reach them. So the baseline run finished about 76 hours after it started, only 8 hours ahead of the cloud runs. None of this changed the results, since every run still saw the same 5 billion tokens. Faster chips on a shared cluster, steady but slower chips in the cloud: that is exactly the trade-off spreading work across providers is meant to absorb.

What the full run will cost in time. The sweep used about 3,000 GPU-hours in total: roughly 2,000 on the cloud A100s and 1,000 on the H200s. At the pace we measured, one pass over the whole 18.6-billion-token harvest from Part 1 would take about 230 hours of training on one 8-GPU H200 machine, or about 310 hours on the A100s: ten to thirteen days, before any time lost along the way. That is why the plan spreads the work across partners rather than waiting for one.

The lesson applies beyond us. Top-tier AI is usually described as something only the largest labs can afford. A single top-tier training run now costs $50 to 80 million. Our whole plan is budgeted to deliver open models for 40 African languages at well under 1% of that, and the only reason that is possible is that it combines donated cloud credits, a partner GPU cluster, and a local grant rather than betting on one. (We are not there yet. This is the plan we are executing, one experiment at a time.) For under-resourced research, compute collaboration is not a nice-to-have. It is the method. Every additional partner is another experiment we can run, another language we can reach. If you have compute to share, that is a direct contribution to closing this gap.

For the public good, and on your terms

A word about why we do this and how we handle what you give us, because both matter.

This is public-good research. AfirkaLLM is an academic project, and everything it produces is released openly for anyone to build on: the corpus, benchmarks, and platform already, and the models as they are trained. The goal is not a product to sell. It is to make sure the languages spoken by hundreds of millions of people are not left out of the technology now reshaping the world. The work belongs to the communities whose languages it serves.

We track every source, and we honour every licence. Where data comes from is not a footnote here. The system enforces it. Every dataset carries its source and its licence, recorded and visible. Data whose licence is unknown or unclear is excluded from training automatically. Nothing untraceable gets into a model. Every document’s language is confirmed by two independent detectors before it is used, every cleaning step is a reviewable, re-runnable script, and every action that matters (every edit, approval and deletion) is written to an audit log.

Data you contribute is treated exactly on the terms it comes with. If a licence restricts a dataset to research, we use it only for research. If it requires attribution, it is attributed. If it forbids redistribution, we do not redistribute it. Contributors and reviewers work under explicit consent, with the right to opt out at any time, and are credited as authors on the resulting datasets and papers. We would rather leave data out than use it in a way its licence, or the person who shared it, did not intend.

That is the promise behind what we ask in the final part of this series: your language, your data, and your terms in service of tools that finally include it.

Next in the series: Part 4. What is missing and how to help, coming in the next few days.

Read about the project and how you can join to contribute: app.afirkallm.org/#join

Sign up and register your language: app.afirkallm.org/apply

🎥 Watch: LLM Building and Sovereign AI. In 2026 we have also been running a special theme of our Data Science for Society Seminar Series focused on LLM building and sovereign AI. Recordings are available on YouTube: watch the LLM Building and Sovereign AI playlist, and subscribe to the channel so you don’t miss new sessions.

AfirkaLLM is a project of the Data Science for Social Impact lab, University of Pretoria. We build open datasets, models, and tools for 40 African languages, and we would love your help doing it.

This work was primarily human-created. AI was used to make stylistic edits, such as changes to structure, wording, and clarity. AI was used to edit content, such as scope, information, and ideas. AI was prompted for its contributions, or AI assistance was enabled. AI-generated content was reviewed and approved. The following model(s) or application(s) were used: Claude, Gemini.