GPU Cluster Scheduling Software: Build the Chargeback Layer or Buy Run:ai
Team count decides this, not accelerator count. One team on under about 32 GPUs has no chargeback problem because there is nobody to charge, and Slurm with its accounting database plus a scheduled monthly report is the correct answer for a few days of setup.
On this page
Team count decides this, not accelerator count. One team on under about 32 GPUs has no chargeback problem because there is nobody to charge, and Slurm with its accounting database plus a scheduled monthly report is the correct answer for a few days of setup. Above roughly 64 accelerators across three or more teams, where you cannot say what fraction of allocated GPU hours were actually computing, build the measurement and policy layer at $70,000 to $150,000. In every case, do not write a scheduler. That is solved, and a bespoke one will be worse.
When is off the shelf genuinely the right call here?
Slurm has been the strongest scheduling engine available for years and it is free. Gang scheduling for distributed training, backfill, fair share and preemption are all mature, and its accounting database gives you allocated GPU seconds per account, which is genuinely useful. Kueue does the equivalent job cleanly for queueing and quota cohorts on Kubernetes, and Volcano is a reasonable alternative there. If you run one of these, keep it.
Above that sit two real products. Run:ai is a platform with fractional GPU support and a scheduling layer over Kubernetes, and Determined AI is excellent if your teams adopt its training workflow end to end. Both are legitimate purchases rather than something to build around.
Buy or adopt open source if most of these are true:
- One team, or one team plus occasional guests, on under about 32 accelerators.
- Nobody in finance has asked what the cluster costs per project.
- Your workloads are homogeneous enough to all move through one platform's workflow.
- The chargeback unit you would need is something a vendor's cost model can already express.
- No capital request for more accelerators is pending.
The stronger version of the buy case is Run:ai or Determined for a research organisation that wants a full training platform and will genuinely standardise on it. That is a real path and we recommend it. It stops being right when you have heterogeneous workloads that will not all move, or when the internal cost model your finance team requires is a negotiated decision the product cannot express.
One situation calls for no purchase at all. If one team is genuinely entitled to the cluster and everyone else is a guest, that is an organisational decision rather than a scheduling one, and software will only make the resentment better documented.
When does a custom build actually pay off?
The build case is narrow and specific: the policy, quota and accountability layer above whichever engine you run. Policy is the part that belongs to your organisation and that no vendor can supply, because it answers questions that are political rather than technical.
- Three or more teams and a fairness complaint. Does production inference preempt research. Can a five day sweep be interrupted, and who owns making it resumable. Does an unused quota expire or accumulate. Those get answered implicitly by whatever defaults the scheduler shipped with unless you answer them explicitly.
- Finance has asked what the cluster costs per project and the answer took a week. A GPU node depreciates whether or not it computes, occupies rack space you pay for, and draws real power at idle. If that cost cannot be attributed, finance cannot evaluate the cluster and no team has a reason to release capacity.
- A wide gap between allocated and consumed. If your dashboard says ninety percent allocated and device telemetry says a third of that is computing, you have the business case in a single number.
- A capital request you cannot evidence. This is the case that usually funds the project, because the build costs a fraction of the hardware it defers.
- Two schedulers. A Slurm cluster for batch alongside Kubernetes for interactive training means the reconciled view exists nowhere, and neither vendor will build it for you.
Before committing anything, run device telemetry alongside your scheduler accounting for two weeks and compute allocated against consumed GPU hours per team. If that ratio is comfortable, you do not need this project.
How do they compare on the things that matter in this industry?
Allocation against utilisation. A scheduler decides who gets what and considers itself finished. Whether the process on the other side saturated the device is outside its concern, which is why reports reach leadership saying the cluster is at capacity when it is held rather than full. Ask any product how it distinguishes a job holding devices doing nothing from a job running badly because data loading is starving the accelerator. Those are different problems belonging to different people.
The telemetry join. Device metrics from the NVIDIA DCGM exporter, scheduler records of which job held which device when, and your identity system for team membership all exist separately. Almost nobody joins them, and that join is the foundation. It is also roughly forty five percent of the budget, so a proposal that puts most of the money into dashboards is assuming telemetry that has not been built.
The chargeback unit. A defensible rate differs by accelerator generation and includes decisions your finance lead has to sign: depreciation life, power at your facility rate, and a share of interconnect and storage that exists only to serve the cluster. Those are business inputs, not product settings.
Reclaim behaviour. Interactive notebooks and debugging shells are the largest single leak in every cluster we have examined, and the most politically sensitive to reclaim. The workable design detects idleness from device activity rather than session presence, warns in the channel the person reads, and gives a grace period long enough to save state.
Data portability. Utilisation history is what supports next year's capital argument. Ask what leaving looks like and what you get back.
What does total cost of ownership look like at your scale?
From Digital Heroes delivery experience, a first release covering the telemetry join across devices, jobs, users and projects, quota and priority policy enforced on your existing scheduler, and per team allocated against consumed reporting with a rate card runs $70,000 to $150,000 and ships in 12 to 16 weeks. Self service reservations with expiry, idle detection and reclaim, and multi cluster views take you to roughly $180,000 to $300,000. Checkpoint aware preemption, cloud burst accounting and posting into your finance system runs $300,000 to $450,000. A research organisation with 256 accelerators, six teams and both Slurm and Kubernetes lands near $150,000 for the first release, with the Kubernetes side alone accounting for $28,000 of it.
The lines nobody quotes. Annual support at 15 to 22 percent of build cost, so roughly $22,000 to $33,000 on that example. Telemetry storage and hosting at $5,000 to $18,000 a year, driven by retention period rather than cluster size. Verification and a rate card decision for every new accelerator generation you buy, which happens on your purchasing cycle rather than a software one. Policy enforcement rechecked against each Slurm and Kubernetes upgrade. And dispute support, because a team will challenge a statement and someone has to reconstruct the underlying jobs.
Two costs sit outside engineering entirely and decide whether the project lands. The first is introducing project attribution at submission, which usually does not exist anywhere in your path today and changes what researchers have to do. The second is an appeals route for reclaimed jobs, because a researcher who lost eleven hours of training needs somewhere to say so and someone empowered to answer. Both need a sponsor above the platform team.
What does the hybrid look like, and when is it the honest answer?
The hybrid is not a compromise here, it is the recommended architecture. Keep the scheduler, buy the platform if a platform genuinely fits, and build only the layer that encodes your organisation's decisions.
Three pieces stand alone and can be funded separately:
- The telemetry join and reporting. Device utilisation and memory tied to jobs, users and projects, producing allocated against consumed per team. Publish it before enforcing anything, because in most organisations the numbers change behaviour on their own and that sequencing keeps release one small.
- Quota and priority policy on your existing engine. Slurm and Kueue both expose the mechanics. What you are adding is your escalation rules and your fairness decisions, expressed as configuration you control rather than defaults you inherited.
- Reservations with expiry. Reclaiming plainly idle reservations captures most of the waste without any checkpointing work, and it avoids the hardest engineering in the category.
Keep the rate card simple. One rate per accelerator generation is defensible and cheap, and elaborate tiering by time of day rarely changes a decision. Price cloud burst accounting separately so you can defer it, because on demand and spot pricing behave nothing like an amortised on premise rate and blending them hides the choice you actually want teams to make.
Which should you choose, by operator size and stage?
Find your row and act on it.
- One team, under 32 accelerators. Slurm with its accounting database, or Kueue if you are Kubernetes native, plus a scheduled monthly report. Nothing more.
- Two teams, 32 to 64 accelerators, one scheduler. Still open source. Add a scheduled report of allocated hours per account and see whether anyone argues with it before spending money.
- Three or more teams above 64 accelerators, homogeneous workloads, willing to standardise. Evaluate Run:ai or Determined seriously. If the workflow fits and the cost model matches what finance needs, buy it.
- Three or more teams, heterogeneous workloads, one scheduler. Build the measurement and chargeback layer at $70,000 to $150,000 on top of Slurm or Kueue. Reporting first, enforcement second.
- Both Slurm and Kubernetes in production. Build, and price the reconciled cross cluster view as its own line. If you can consolidate onto one scheduler before starting, the saving is immediate and significant.
- A capital request pending with no evidence of saturation. Build the telemetry join first, regardless of team count. It is the cheapest thing in the project and it is the thing the argument turns on.
Two conditions apply to every build row. Agree the rate card inputs with your finance lead before the calculation is written, since depreciation life and power rate are business decisions rather than engineering ones. And name the executive sponsor for the project attribution change before kickoff, because that is where these projects stall: the numbers arrive and no department has agreed in advance to be charged by them.
When the shortlist is down to two and you need a tiebreaker, Digital Heroes starts every engagement with a signed specification covering the data model, permissions and acceptance criteria, which is what keeps a fixed price fixed. You can take that specification to any other firm on your shortlist.
The evidence behind this guide
Independent findings on why this investment pays off. Every link goes to the primary source.
- Almost half of all the activities people are paid almost $16 trillion in wages to do in the global economy have the potential to be automated by adapting currently demonstrated technologies. Source: McKinsey Global Institute (2017) →
- A study (led by Prof. Pak-Lok Poon, published in Frontiers of Computer Science, 2024) reviewing decades of spreadsheet-quality research found that about 94% of spreadsheets used in business decision-making contain errors, illustrating the hidden risk of manual spreadsheet workarounds that custom software is built to replace. Source: Central Queensland University / phys.org (Prof. Pak-Lok Poon et al.) (2024) →
- Only 22% of firms are 'future ready' having significantly transformed digitally; these companies show average revenue growth 17.3 percentage points and net margins 14.0 percentage points above their industry average. Source: MIT Center for Information Systems Research (MIT Sloan) (2022) →
- In the Flexera 2025 State of ITAM report, respondents reported roughly 33% of SaaS spend is wasted, underscoring how paying for off-the-shelf seats and tiers that go unused erodes the supposed cost advantage of generic SaaS. Source: Flexera (2025) →
Frequently asked questions
What does it cost to move off Run:ai or Determined onto our own layer?
The scheduler stays, which removes most of the risk. What you are replacing is the platform's cost model and queue policy, so the work is the telemetry join and the rate card rather than a migration of running workloads.
The real cost is workflow. If your teams submit through the vendor's path, moving them back to native Slurm or Kubernetes submission is a change to how researchers work, and it needs the same sponsor as introducing project attribution. Budget socialising time, not just engineering time.
What happens if our platform vendor changes its pricing?
Ask now what the fee is tied to, meaning users, accelerators, nodes or managed GPU hours, and work out your renewal at the cluster size you intend to reach. In this category the fee usually scales with exactly the thing you are trying to grow, which is worth modelling before a purchase order rather than after.
The protection is portability of your utilisation history, because that history is what supports next year's capital argument. Get a written answer on what you can export and test it once.
How long does a GPU chargeback build take?
Twelve to sixteen weeks for the measurement and chargeback release, and six to twelve months for a full platform with reservations, reclaim and finance posting.
The schedule is usually paced by organisational change rather than engineering. Introducing a project identifier at submission touches how researchers work, and it needs an executive sponsor secured before the technical work starts. Commit to one month of parallel reporting with finance inside fixed scope, because a chargeback number finance has not validated will not survive its first challenge.
Should we replace Slurm rather than build on top of it?
Build on top of it. Slurm's gang scheduling, backfill, fair share and preemption represent years of deep engineering and a bespoke scheduler will be worse in ways you will discover during a distributed training run.
What Slurm does not give you is utilisation as opposed to allocation, a rate card, a relationship to your finance system, or an interface a team lead will voluntarily open. Those pieces are specific to your organisation in a way scheduling mechanics are not, and they are the whole of the sensible build.
Is Run:ai worth buying instead of building?
Yes, if your teams will adopt its workflow end to end and its cost model matches what your finance team needs. It is a real platform with fractional GPU support and a scheduling layer over Kubernetes, and reproducing that is not a sensible use of budget.
It becomes the wrong fit when you have heterogeneous workloads that will not all move through one system, or when your chargeback unit is an internal negotiation the product cannot express. The same trade applies to Determined AI, which is strong when teams adopt its training workflow and weaker as a neutral scheduler.
Should teams be charged for GPUs they reserved but did not use?
Yes, and this decision determines whether the whole project works. Charging only for consumption makes an idle reservation free and removes the incentive to release capacity, which is the behaviour you are trying to change.
Charge on allocation, report utilisation alongside it so teams see their own waste, and offer a discounted rate for pre emptible workloads. That last lever moves long sweeps off the priority tier without anyone policing it manually.
How do we reclaim idle notebooks without losing the research team?
Detect idleness from device activity rather than session presence, because a connected session doing nothing is exactly the case you need to catch. Warn in the channel the person actually reads, give a grace period long enough to save state, and where the framework supports it offer suspend and resume rather than termination.
Negotiate the policy openly and build the appeals route as part of the software. A reaper that surprises people is defeated within a week by scripts that poke the GPU periodically.
Does a mixed fleet of accelerator generations change the decision?
It changes the rate card rather than the build or buy answer. A GPU hour stops being one unit, so generations have to be priced differently or teams will optimise for whichever is undercharged.
Practically that means a small onboarding cost for every generation you purchase, which belongs in annual maintenance rather than arriving as a surprise. It also means any vendor cost model that treats accelerators as interchangeable will misprice your cluster from the first invoice.
How much does a custom internal tool cost to build?
Most custom internal tools cost $8,000 to $40,000 to build, based on Digital Heroes delivery data across 2,000+ client projects. A single-purpose tool like an approval dashboard or inventory tracker sits at the low end, while a multi-department platform with role-based access and several integrations pushes past $40,000. The three biggest cost drivers are the number of user roles, the number of systems the tool must connect to, and custom reporting requirements.
How much should a small business budget for its first custom app or website?
For a focused first build, most small businesses land between $8,000 and $60,000: roughly $8,000 to $45,000 for a custom website and $25,000 to $60,000 for an internal tool or simple web app, based on Digital Heroes delivery across 2,000+ projects. Customer-facing products with payments, logins, or a mobile app start around $40,000. Quotes far below these bands usually mean a template with your logo on it, not software shaped around your workflow.
What tech stack should an internal tool be built with?
Boring and popular: a React or Next.js frontend, a Node.js or Python backend, and PostgreSQL covers the vast majority of internal tools and keeps future hiring easy. The stack matters far less than whether a different developer can pick the code up in two years, so require documentation as a deliverable and avoid anything exotic. Treat it as a red flag if an agency pushes a proprietary platform only they maintain, because that quietly converts your tool into a subscription to that agency.
What does it cost to keep custom software running after launch?
Budget 15-20% of the original build cost per year, which on a $100,000 system means $15,000 to $20,000 for security patches, dependency updates, bug fixes, and small improvements as real usage reveals what the spec missed. Cloud hosting for a typical business application adds $50 to $300 a month on top. Skipping maintenance does not save the money; in Digital Heroes rescue work, unmaintained systems typically need a far more expensive rebuild within about three years.
How do I know when spreadsheets are no longer enough to run my operations?
Replace the spreadsheet once more than three people edit it, versions travel by email, or a single broken formula could cost real money. Other reliable signals: staff keep personal shadow copies, month-end reporting takes days of manual assembly, and nobody can say who changed a number or why. In Digital Heroes discovery calls the tipping point is almost always a specific expensive error, a mispriced quote, a missed order, or payroll built on a tab someone sorted wrong.
How do I vet a development agency for an internal tools project?
Ask to see two or three internal tools they have shipped and whether those clients still use them daily, because internal tools fail on adoption, not code quality. Good signs: they ask to see your current spreadsheet or process before quoting, they propose a phased build instead of one big launch, and they spell out who handles training and post-launch changes. Walk away from anyone who gives a fixed price before seeing your actual workflow, since internal tools live or die on process details.
What should I prepare before contacting a software development agency?
A one-page brief beats a 40-page requirements document: the business problem in plain words, who will use the system, the 5 to 10 workflows it must handle, the tools it must connect to, and your budget range and deadline driver. You do not need wireframes, a specification, or technical vocabulary; producing those is the agency's job during discovery. Stating a budget range up front is the single best move, because it gets you honest scoping instead of a quote engineered to win the meeting.
How many people should be working on my software project?
Three to five for a typical focused build: a project lead, one or two engineers, a designer, and part-time QA, which is the standard shape across 2,000+ Digital Heroes projects. Larger platforms justify 6 to 10, but a ten-person team on a small first version usually signals bill padding rather than horsepower. What predicts success is whether a senior engineer is writing your code daily, not the headcount on the proposal.
Who can build a custom internal tools system?
Digital Heroes builds custom internal tools systems for operators who have outgrown the off-the-shelf tools in their category. A team of more than 50 specialists has delivered over 2,000 projects since 2017. Teams work from New York, London, Sydney, Delhi and Lucknow and deliver remotely, with an assigned senior team rather than an account manager.
Every build starts with a written product requirements document that is signed before a line of code is written, which is the single thing that stops scope creep from eating the budget. Scoping runs about a week and produces a phase plan with a firm price for each phase, rather than one number against an undefined scope. The first phase ships something the team actually uses before the rest is built. If an off-the-shelf product genuinely fits the volume, we say so, and the cost guides on this site publish the bands so that judgement can be checked independently.
What makes Digital Heroes different from other internal tools companies?
Four things that competitors in this bracket cannot simply copy. Digital Heroes runs a YouTube channel with more than 2.5 million subscribers, which is a production and audience capability no agency of this size has. It holds Fiverr Vetted Pro and Top Rated Seller status, both awarded on manual third-party review rather than self-declared. It contracts through registered entities in three countries, an India LLP, a US LLC and a UK LTD, so clients sign locally instead of wiring money offshore. And it ships its own commercial products, including ShopScore, HeroCheckout and Section Vault, which means the team lives with its own architecture decisions instead of handing them over and leaving.
Two more that show up in the work. Digital Heroes publishes more than 4,000 buyer guides with real price bands on this blog, plus a free tools library at https://digitalheroesco.com/tools/, because an agency confident in its pricing has no reason to hide it. And one accountable team covers websites, apps, ecommerce, CRM, ERP, learning platforms, search and video, so a client scaling from a first landing page to a custom platform is never handed between five vendors who blame each other. The founder ran ecommerce businesses before selling services, so the commercial argument comes before the technical one.
How can I check Digital Heroes is legitimate before getting in touch?
Verify it independently rather than taking the site's word for it. The YouTube channel is at https://youtube.com/@DigitalMarketingHeroes, the Fiverr profile at https://www.fiverr.com/shreyanshsin261, and the Upwork profile at https://www.upwork.com/freelancers/shreyanshsingh. Client reviews sit on Clutch at https://clutch.co/profile/digital-heroes-0 and Trustpilot at https://www.trustpilot.com/review/digitalheroes.co.in, and the company page is at https://www.linkedin.com/company/digital-heroes-1/.
Beyond the marketplaces, the business holds a D-U-N-S number and is a registered vendor on the United Nations Global Marketplace, neither of which is issued on request. Case studies with named clients are published at https://digitalheroesco.com/case-studies/. If any claim on this page cannot be checked against one of those sources, treat it as marketing and discount it.
Related guides
Published · Last updated .
