Worker & Autoscaling Integration

Driving NKS worker capacity from Go touches three things the API reference documents individually but never connects: every mutating call returns an operation rather than a resource, some node pool fields cannot be changed after creation, and scaling down by hand is not the same as letting the autoscaler do it. Each of those is cheap to discover the hard way and expensive to get wrong in a long-running service.

The code below is a single package, compiled against nirvana-go v1.94.2 with go build and go vet.

Imports

import (
  "context"
  "errors"
  "fmt"
  "net/http"
  "time"

  "github.com/nirvana-labs/nirvana-go"
  "github.com/nirvana-labs/nirvana-go/compute"
  "github.com/nirvana-labs/nirvana-go/nks"
  "github.com/nirvana-labs/nirvana-go/operations"
  "github.com/nirvana-labs/nirvana-go/option"
  "github.com/nirvana-labs/nirvana-go/shared"
)

Client Setup

A worker builds one client and reuses it. Credentials come from the environment, so nothing has to be threaded through your config: NIRVANA_LABS_API_KEY and NIRVANA_LABS_BASE_URL are read by nirvana.NewClient automatically.

// NewClient builds a client for a long-running worker. NIRVANA_LABS_API_KEY and
// NIRVANA_LABS_BASE_URL are read from the environment automatically; pass
// option.WithAPIKey to override.
func NewClient() nirvana.Client {
  return nirvana.NewClient(
    option.WithMaxRetries(3),
  )
}

Requests have no default timeout. Bound them with a context rather than relying on the client, and use option.WithRequestTimeout if you need a per-retry bound instead of a whole-lifecycle one.

The Operation Pattern

This is the part that causes trial-and-error edits. Clusters.New, Pools.New, Pools.Update and Pools.Delete all return an *operations.Operation whose Status starts as pending. The resource is not usable yet, and its ID is on op.ResourceID — not op.ID, which identifies the operation itself.

There is no built-in waiter in the SDK, so a service needs its own:

// ErrOperationFailed is returned when the API reports an operation as failed.
var ErrOperationFailed = errors.New("nirvana: operation failed")

// WaitForOperation polls an operation until it reaches a terminal state.
//
// The SDK has no built-in waiter: every mutating call returns an *Operation
// whose status starts at "pending", so a caller that treats the returned value
// as the finished resource will act on state that does not exist yet.
func WaitForOperation(ctx context.Context, client nirvana.Client, operationID string) (*operations.Operation, error) {
  const interval = 5 * time.Second

  ticker := time.NewTicker(interval)
  defer ticker.Stop()

  for {
    op, err := client.Operations.Get(ctx, operationID)
    if err != nil {
      return nil, fmt.Errorf("polling operation %s: %w", operationID, err)
    }

    switch op.Status {
    case operations.OperationStatusDone:
      return op, nil
    case operations.OperationStatusFailed:
      return op, fmt.Errorf("%w: %s %s (operation %s)", ErrOperationFailed, op.Type, op.Kind, op.ID)
    case operations.OperationStatusPending, operations.OperationStatusRunning, operations.OperationStatusUnknown:
      // keep waiting
    }

    select {
    case <-ctx.Done():
      return nil, ctx.Err()
    case <-ticker.C:
    }
  }
}

Two details worth copying rather than re-deriving:

• op.ResourceID vs op.ID. Poll with op.ID; read the created resource with op.ResourceID. Mixing them up produces a 404 that looks like the resource was never created.
• Treat unknown as non-terminal. OperationStatus has five values — pending, running, done, failed, unknown — and only done and failed are terminal. Exiting the loop on anything else strands the caller.

Creating a Cluster with Autoscaling

Autoscaling is a required field on cluster create, not an optional one, and it is a cluster-level toggle rather than a per-pool setting.

// CreateAutoscalingCluster creates a cluster with platform autoscaling enabled
// and blocks until it is ready. A cold cluster can take ~20 minutes, so give
// the context a generous deadline.
func CreateAutoscalingCluster(ctx context.Context, client nirvana.Client, projectID, vpcID, name, k8sVersion string) (string, error) {
  op, err := client.NKS.Clusters.New(ctx, nks.ClusterNewParams{
    Autoscaling:       true,
    KubernetesVersion: k8sVersion,
    Name:              name,
    ProjectID:         projectID,
    Region:            shared.RegionNameUsSva2,
    VPCID:             vpcID,
    Tags:              []string{"production", "workers"},
  })
  if err != nil {
    return "", fmt.Errorf("creating cluster: %w", err)
  }

  if _, err := WaitForOperation(ctx, client, op.ID); err != nil {
    return "", err
  }
  return op.ResourceID, nil
}

KubernetesVersion is required and changing it later recreates the cluster. List the available versions with client.NKS.KubernetesVersions.List rather than pinning a string you guessed.

Declaring a Worker Pool

// CreateWorkerPool declares a worker pool. instance_type, labels and taints are
// fixed at creation: the update endpoint accepts only name, node_count and tags,
// so changing a pool's shape means creating a new pool and draining the old one.
func CreateWorkerPool(ctx context.Context, client nirvana.Client, clusterID string) (string, error) {
  op, err := client.NKS.Clusters.Pools.New(ctx, clusterID, nks.ClusterPoolNewParams{
    Name:      "compute",
    NodeCount: nirvana.Int(1),
    NodeConfig: nks.NKSNodePoolNodeConfigParam{
      InstanceType: "n1-highcpu-16",
      BootVolume: nks.NKSNodePoolBootVolumeParam{
        Size: 128,
        Type: compute.VolumeTypeABS,
      },
      Labels: []string{"workload=batch"},
      Taints: []string{"dedicated=batch:NoSchedule"},
    },
    Tags: []string{"production", "workers"},
  })
  if err != nil {
    return "", fmt.Errorf("creating node pool: %w", err)
  }

  if _, err := WaitForOperation(ctx, client, op.ID); err != nil {
    return "", err
  }
  return op.ResourceID, nil
}

Caution

instance_type, labels and taints are immutable after pool creation. The update endpoint accepts only name, node_count and tags, so a service that wants to change a pool’s shape has to create a new pool and drain the old one — there is no in-place edit and no error will tell you this until the change is silently absent.

Labels and taints are flat []string, not maps or structs: labels are "key=value" and taints are "key=value:Effect", where Effect is NoSchedule, PreferNoSchedule or NoExecute. Keys under kubernetes.io, k8s.io and nirvanalabs.io are reserved.

Boot volume size is bounded at 64–512 GB, and NodeCount accepts 0–100. A pool created with NodeCount: 0 is a scale-from-zero pool: it stays empty until a pod that tolerates its taints is pending, which is how you make an instance type available to the autoscaler without paying for an idle node.

Note

Do not set every pool to zero. The cluster needs at least one worker to provision and to host CoreDNS, so keep one baseline pool at NodeCount >= 1.

Scaling a Pool

// ScalePool sets a pool's node count. Prefer letting cluster autoscaling do
// this: a manual scale-down terminates nodes without draining pods.
func ScalePool(ctx context.Context, client nirvana.Client, clusterID, poolID string, desired int64) error {
  op, err := client.NKS.Clusters.Pools.Update(ctx, clusterID, poolID, nks.ClusterPoolUpdateParams{
    NodeCount: nirvana.Int(desired),
  })
  if err != nil {
    return fmt.Errorf("scaling pool %s to %d: %w", poolID, desired, err)
  }

  _, err = WaitForOperation(ctx, client, op.ID)
  return err
}

Caution

Prefer autoscaling for scale-down. Shrinking a pool by lowering NodeCount is currently disruptive however you do it — the platform picks a worker and terminates it without cordon, drain, PodDisruptionBudget respect, or volume detach, so anything running on that node is killed abruptly. The autoscaler, by contrast, cordons and evicts gracefully. If a service must shrink a pool itself, kubectl drain the affected nodes first.

Scaling up by hand is safe, and is the right move when you know demand is coming before the autoscaler could observe it.

Reading Current Capacity

A scheduler needs a snapshot of what exists. ListAutoPaging follows the cursor for you, so there is no pagination loop to write:

// PoolCapacity is a snapshot of one pool, the shape a scheduler needs.
type PoolCapacity struct {
  ID           string
  Name         string
  InstanceType string
  NodeCount    int64
  Ready        bool
}

// ObserveCapacity lists every pool on a cluster, following pagination.
func ObserveCapacity(ctx context.Context, client nirvana.Client, clusterID string) ([]PoolCapacity, error) {
  var out []PoolCapacity

  pager := client.NKS.Clusters.Pools.ListAutoPaging(ctx, clusterID, nks.ClusterPoolListParams{})
  for pager.Next() {
    pool := pager.Current()
    out = append(out, PoolCapacity{
      ID:           pool.ID,
      Name:         pool.Name,
      InstanceType: pool.NodeConfig.InstanceType,
      NodeCount:    pool.NodeCount,
      Ready:        pool.Status == shared.ResourceStatusReady,
    })
  }
  if err := pager.Err(); err != nil {
    return nil, fmt.Errorf("listing pools for cluster %s: %w", clusterID, err)
  }
  return out, nil
}

Check pager.Err() after the loop. Next() returning false means either “finished” or “failed”, and without the error check a transport failure looks like an empty cluster — which, in an autoscaler, reads as “scale everything up”.

Distinguishing Rejections from Outages

A retry loop needs to know whether the API responded at all, and if it did, whether the failure was the request’s fault or the service’s. Anything the API responded to surfaces as *nirvana.Error carrying the status code; anything else is a transport problem.

// ClassifyError separates a response the API produced from a transport failure.
// It reports true for any status the API returned, so the caller must read the
// code -- see Retryable -- rather than treating every status as a rejection.
func ClassifyError(err error) (statusCode int, isAPIError bool) {
  var apiErr *nirvana.Error
  if errors.As(err, &apiErr) {
    return apiErr.StatusCode, true
  }
  return 0, false
}

// Retryable mirrors the client's own retry policy: a timeout, a conflict, a
// rate limit, or any server error. Seeing one of these means the condition
// outlived the client's retry budget, not that the request is wrong -- so the
// answer is a longer backoff, never an edit to a valid request. Keep this in
// step with option.WithMaxRetries; the client applies the same set internally.
func Retryable(statusCode int) bool {
  switch statusCode {
  case http.StatusRequestTimeout, http.StatusConflict, http.StatusTooManyRequests:
    return true
  }
  return statusCode >= http.StatusInternalServerError
}

isAPIError on its own is not a verdict. Every status sets it, so branching on it alone reports a rate limit or a service outage as a rejected request — and tells whoever is on call to fix a request that was never the problem.

Nor is 4xx the dividing line. 408, 409 and 429 are 4xx yet temporary, which is why the client retries them; a rule like code < 500 sorts them with 400 and throws away a request that only needed more time.

The client already retries those cases before an error reaches you — with option.WithMaxRetries(3) above, up to three times, plus connection errors — using exponential backoff that honours a Retry-After header. So a 429 or 503 arriving in your code means that budget is spent. Reissuing it straight away just burns a fresh budget against the same condition; the response that works is a longer horizon, usually the next reconcile pass.

Caution

ClassifyError does not recognise an operation failure. ErrOperationFailed comes from polling, not from a rejected HTTP request, so there is no *nirvana.Error in its chain and errors.As finds nothing. Check ErrOperationFailed before calling ClassifyError, or a create that the platform accepted and then failed asynchronously gets reported as “could not reach the API” — which in an autoscaler means alerting on a network outage that never happened, and retrying a request that already ran.

There are five outcomes worth separating, and they need checking in this order:

Outcome	Detect with	What it means
The operation ran and failed	errors.Is(err, ErrOperationFailed)	The platform accepted the request and could not complete it. Do not retry blindly; inspect the resource
You ran out of time	errors.Is(err, context.DeadlineExceeded)	Still possibly running server-side. Poll again rather than reissuing the create
The failure is temporary	Retryable(code) — 408, 409, 429, any 5xx	A timeout, conflict, rate limit or server fault that outlived the client’s retries. Back off further; the request is fine
The API refused the request	any other status	Fix the request; retrying is pointless
The API was unreachable	none of the above	Transport failure after the client’s own retries. Back off and retry

Putting It Together

package main

import (
  "context"
  "errors"
  "log"
  "time"

  "gocheck/nksworkers"
)

// fatal reports an error with the distinction that matters operationally:
// the platform ran the operation and it failed, we ran out of time, the
// condition is temporary, the API refused the request, or we could not reach
// it at all. Check ErrOperationFailed first -- it is wrapped, so ClassifyError
// does not see it and would otherwise misreport an async failure as an outage.
func fatal(action string, err error) {
  switch {
  case errors.Is(err, nksworkers.ErrOperationFailed):
    log.Fatalf("%s: the platform ran the operation and it failed: %v", action, err)
  case errors.Is(err, context.DeadlineExceeded), errors.Is(err, context.Canceled):
    log.Fatalf("%s: gave up waiting; the operation may still be running: %v", action, err)
  default:
    // An API error carries a status code, and the code decides the
    // meaning. Only a status outside the retryable set blames the request.
    code, isAPI := nksworkers.ClassifyError(err)
    switch {
    case isAPI && nksworkers.Retryable(code):
      log.Fatalf("%s: the API returned %d and the client's retries are already spent; "+
        "the request is valid, so back off and try later rather than changing it: %v", action, code, err)
    case isAPI:
      log.Fatalf("%s: the API rejected the request with %d; fix it, retrying will not help: %v", action, code, err)
    default:
      log.Fatalf("%s: could not reach the API: %v", action, err)
    }
  }
}

func main() {
  client := nksworkers.NewClient()

  // Cluster and pool creation are slow; bound them with a context, not a
  // fixed sleep.
  ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
  defer cancel()

  clusterID, err := nksworkers.CreateAutoscalingCluster(
    ctx, client,
    "123e4567-e89b-12d3-a456-426614174000", // project_id
    "123e4567-e89b-12d3-a456-426614174001", // vpc_id
    "production-workers",
    "v1.34.4",
  )
  if err != nil {
    fatal("creating cluster", err)
  }

  poolID, err := nksworkers.CreateWorkerPool(ctx, client, clusterID)
  if err != nil {
    fatal("creating worker pool", err)
  }

  pools, err := nksworkers.ObserveCapacity(ctx, client, clusterID)
  if err != nil {
    fatal("observing capacity", err)
  }
  for _, p := range pools {
    log.Printf("pool %s (%s): %d nodes, ready=%t", p.Name, p.InstanceType, p.NodeCount, p.Ready)
  }

  // Only scale manually when autoscaling is off, or when growing a pool.
  if err := nksworkers.ScalePool(ctx, client, clusterID, poolID, 5); err != nil {
    fatal("scaling pool", err)
  }
}

Related

• Autoscaling
• Node Pools
• Instance Types
• SDKs & Tools
• nirvana-go on pkg.go.dev