[NOTES] Kubernetes vs SLURM: Scheduler Architecture Notes for 2026

18 Jan, 2026

Introduction

When running large-scale distributed workloads, the default Kubernetes scheduler often falls short of specialized requirements. High-performance computing (HPC) environments frequently demand sophisticated scheduling strategies that Kubernetes’ built-in components cannot provide. This is where custom schedulers like those inspired by SLURM (Simple Linux Utility for Resource Management) come into play, offering fine-grained control over job placement, resource allocation, and workload management.

In this post, we’ll explore how to implement custom schedulers in Kubernetes, the lessons learned from SLURM’s approach, and practical patterns for integrating advanced scheduling logic into your Kubernetes clusters.

Understanding Default Kubernetes Scheduling

Before diving into custom schedulers, it’s important to understand Kubernetes’ default scheduling model. The built-in kube-scheduler uses a series of filter and score functions to place pods on nodes. It considers factors like:

Resource requests and limits

• Node affinity and pod affinity rules
• Taints and tolerations
• Priority and preemption policies While powerful, this approach treats scheduling as a relatively simple problem: fit the pod on a suitable node. It doesn’t handle complex scenarios that HPC environments require, such as:

Gang scheduling (scheduling multiple pods together atomically)

• Queue-based fair-share resource allocation
• Priority-based job preemption with backfilling
• Advanced topology awareness
• Heterogeneous resource types

What SLURM Brings to the Table

SLURM has been the de facto workload manager in HPC environments for decades. Its scheduler implements several features that make it attractive for complex workload management:

Gang Scheduling: SLURM can guarantee that all tasks in a job start together. This is critical for parallel applications that need synchronized execution.

Fair-Share Scheduling: Resources can be allocated fairly across different users and groups, with the ability to prioritize certain workloads.

Advanced Job Prioritization: SLURM uses a sophisticated priority system that considers job age, fair-share allocation, and explicit priority levels to determine scheduling order.

Backfilling: The scheduler can opportunistically run smaller jobs in gaps left by larger jobs, improving overall cluster utilization.

Topology-Aware Scheduling: SLURM is aware of node topology and can schedule tasks to optimize communication patterns.

Custom Schedulers in Kubernetes

Kubernetes provides several mechanisms for implementing custom scheduling logic:

1. Extending the Default Scheduler

The simplest approach is to use scheduler plugins (introduced in Kubernetes 1.16). These plugins allow you to:

Add custom filter functions to remove unsuitable nodes

• Implement custom scoring functions to rank nodes
• Control reservation and binding behavior Example plugin hooks include:

PreFilter: Early filtering based on pod requirements

• Filter: Eliminate nodes that cannot accommodate the pod
• Score: Rank remaining nodes by suitability
• Reserve: Temporarily allocate resources
• Bind: Assign the pod to the selected node

2. Multiple Schedulers

For more complex requirements, you can run multiple schedulers alongside the default one. Pods can specify which scheduler should handle their placement using the schedulerName field:

apiVersion: v1
Kind: Pod
Metadata:
  Name: my-pod
Spec:
  schedulerName: custom-scheduler
  Containers:
Name: app
    Image: myapp:latest

This allows different schedulers to manage different workload types within the same cluster.

3. Custom Scheduler Frameworks

Several projects have created complete custom schedulers for Kubernetes:

Volcano: A batch scheduling system designed for HPC and big data workloads, featuring gang scheduling, fair-share allocation, and queue management

• Kube-Batch: A batch scheduler supporting job-level scheduling and reclaim strategies
• Karmada: A multi-cluster scheduler for distributing workloads across clusters
• Yunikorn: Apache YuniKorn, a universal resource orchestrator that brings enterprise scheduling capabilities

Building a SLURM-Inspired Scheduler

Here’s a conceptual architecture for a SLURM-inspired Kubernetes scheduler:

Key Components

Job Queue Manager: Manages incoming jobs in queues with different priorities, similar to SLURM’s job queue system.

Priority Calculator: Determines job priority based on: Job age (longer waiting jobs get higher priority)

• Fair-share allocation (ensuring equitable resource access)
• Explicit priority levels
• QoS (Quality of Service) tier Scheduling Algorithm: Select highest-priority job from queue

2. Attempt to schedule all replicas/pods of the job together (gang scheduling)
3. If feasible, allocate resources and create pods
4. If not feasible, check for backfill opportunities
5. Schedule smaller jobs in available slots Resource Monitor: Continuously tracks: Node capacity and utilization

• Pod resource requests and actual usage
• Allocation fairness across users/groups

Implementation Considerations

State Management: Use Kubernetes ConfigMaps or CRDs to store scheduler state (job queues, fair-share data, reservations). Consider using etcd directly for high-performance scenarios.

Communication: Implement webhooks or use the scheduler framework API to receive pod scheduling requests and communicate decisions.

Resource Guarantees: Use resource quotas and priority classes to enforce resource allocations across users and job priorities.

Monitoring and Observability: Instrument the scheduler to emit metrics on: Scheduling latency

• Queue depth and wait times
• Backfill efficiency
• Fair-share allocation accuracy

Integration Patterns

Pattern 1: Coexistence with Default Scheduler

Run a custom scheduler for batch workloads while the default scheduler handles interactive and best-effort workloads. Use selectors and namespace-level defaults to route pods to the appropriate scheduler.

Pattern 2: Hierarchical Scheduling

Implement multiple tiers of schedulers: Global scheduler for cluster-wide resource allocation

• Local schedulers for namespace or tenant-specific logic
• Specialized schedulers for GPU, memory-optimized, or network-intensive workloads

Pattern 3: Preemption and Backfilling

Implement fair-share allocation with dynamic preemption: High-priority jobs can preempt lower-priority ones

• Preempted jobs return to the queue for rescheduling
• Backfill algorithm fills gaps with smaller jobs

Practical Challenges and Solutions

Challenge 1: Gang Scheduling Complexity

Gang scheduling in Kubernetes is non-trivial because pod creation is asynchronous. Solution: Use JobSet (Kubernetes 1.30+) or implement your own batch job CRD that coordinates multiple pods, ensuring atomic scheduling or rollback.

Challenge 2: Topology Awareness

Kubernetes doesn’t have built-in NUMA or advanced topology awareness. Solution: Use Pod Topology Spread Constraints combined with node labels to encode topology information.

Challenge 3: Fair-Share Implementation

Implementing accurate fair-share requires fine-grained accounting. Solution: Track resource consumption per user/group, adjust priority scores dynamically based on cumulative consumption, and use preemption policies to enforce allocations.

Challenge 4: Overcommitment and Overbooking

HPC systems often overbook resources. Solution: Use request/limit separation carefully, implement admission webhooks to enforce overcommitment policies, and use PodDisruptionBudgets to manage graceful eviction.

Emerging Standards

The Kubernetes community is investing in standardized scheduling components:

Karpenter: Autoscaling and bin-packing optimized for containerized workloads

• Gatekeeper: Policy enforcement that can integrate with scheduling decisions
• Resource Slices: Work-in-progress APIs for better resource model representation

Conclusion

Custom schedulers in Kubernetes bring HPC-style workload management to containerized environments. While the default scheduler is excellent for general-purpose workloads, specialized applications benefit from SLURM-inspired features like gang scheduling, fair-share allocation, and sophisticated priority management.

The path forward depends on your specific requirements: if you need basic extensions, scheduler plugins may suffice. For complex batch workloads, consider Volcano or similar frameworks. For organizations building entirely new platforms, implementing a custom scheduler provides maximum flexibility at the cost of additional complexity.

As Kubernetes continues to evolve, we expect deeper integration of batch scheduling concepts into the core platform, potentially reducing the need for external solutions in the future.

Flashcards!

I ask Claude to make flashcards for my tech notes! You should try it too!

Critique

The blog is open for critique. Drop me an email at shivbhosale97@gmail.com with subject: "blog critique" and I will add your points in this section. Also please point out any inaccuracies / mistakes in the essay/post. Your critique will appear in-line like this:

#distributed-systems #gang-scheduling #hpc #infrastructure #kubernetes #kubernetes-at-scale #notes #slurm