Sleep Mode For Spaces

Loft provides two powerful features to reduce Kubernetes cost:

• Sleep Mode which puts namespaces to sleep when nobody is using them, i.e. purging all pods while keeping all resources inside the namespaces during periods of inactivity
• Auto-Delete which deletes namespaces that have been idle for a while

Both of these feature typically rely on Loft's inactivity detection.

Working with Sleep Mode

With sleep mode, you can put Kubernetes namespaces to sleep which means that Loft will set replicas: 0 for all replica-controlled resources such as Deployments and StatefulSets. This means that Kubernetes will delete all pods but the entire configuration of resources within the namespace is still there.

Sleep mode can be:

• Invoked manually
• Triggered by an inactivity timeout (no one has ran a kubectl command in this namespace for X minutes)
• Scheduled using a CRON syntax

Manual + Automatic Sleep / Wake-up

Start Sleep (manual)

UI

1. In the Spaces view, hover over the row of the space that you want to put to sleep
2. While hovering over the row, you will see buttons appear on the right in the Actions column
3. Click on the button to put the space to Sleep
4. Notice how the Status column shows that the space is now sleeping.

CLI

To put a space to sleep using Loft CLI, run:

loft sleep space [name]

Automatic Wakeup

Note that the space will automatically wake up again, once you run a kubectl command within the space.

Automatic Sleep Mode (individual space)

1. In the Spaces view, hover over space that you want to configure automatic sleep mode for
2. While hovering over the row, you will see buttons appear on the right in the Actions column
3. Click on the button to Edit the space
4. In the drawer that appears on the right, expand the Sleep Mode section
5. Use the Sleep After Inactivity field to specify the Time (in minutes) to wait before putting the space to sleep if there is no more user activity in this namespace
6. On the very bottom, click on the button to save the changes

Wake up space

UI

1. In the Spaces view, hover over the Status column of the space that you want to put to sleep
2. While hovering over the row, you will see a tooltip appear that provide information about the sleep state of this space
3. Click on the button to wakeup the space
4. Notice how the Status column shows that the space is now Active again.

CLI

To wake up a sleeping space using Loft CLI, run:

loft wakeup space [name]

kubectl

To wake up a sleeping space using kubectl, run:

kubectl get pods   # or any other kubectl command within the namespace

Scheduled Sleep / Wake-up

Scheduled Sleep & Wake-Up (individual space)

1. In the Spaces view, hover over space that you want to configure automatic sleep mode for
2. While hovering over the row, you will see buttons appear on the right in the Actions column
3. Click on the button to Edit the space
4. In the drawer that appears on the right, expand the Sleep Mode section
5. Expand the Sleep & Wake-Up Schedule section
6. Use the Sleep Schedule field and/or the Wake-Up Schedule field to specify the Conjob Times when the respective namespace should be put to sleep or woken up
7. On the very bottom, click on the button to save the changes

Enforce Sleep Settings

Enforce Sleep Mode For All Spaces Created By User/Team

1. Go to the Clusters view using the menu on the left
2. Switch to the Space Constraints tab

3. Option A: Hover over the space constraints object that you want to configure automatic sleep mode with and click on the button to Edit an existing space constraints object

   Option B: Click the button to create a new space constraints object

4. In the drawer that appears on the right, expand the Enforce Space Settings section
5. Use the Sleep After Inactivity field to specify the Time (in minutes) to wait before putting the space to sleep if there is no more user activity in this namespace
6. On the very bottom, click on the or button to save the changes
7. Switch to the Cluster Access tab
8. Hover over the cluster access of the user or team that you want to configure automatic sleep mode for and click on the button to Edit the cluster access
9. In the drawer that appears on the right, expand the Restrictions section
10. Use the Enforce Space Constraints field to select the Space Constraint you edited or created in Step 3 above
11. On the very bottom, click on the button to update the cluster access

Test with Impersonation

After following the steps above, all spaces created using the cluster access in step 7 will now enforce sleep mode. You can test this behavior by impersonating a user that uses this cluster access.

Working with Auto-Delete

Loft lets you configure an auto-delete for namespaces that have not been used for a certain period of time (inactivity).

Configure Auto-Delete Timeout (individual space)

1. In the Spaces view, hover over space that you want to configure auto-delete for
2. While hovering over the row, you will see buttons appear on the right in the Actions column
3. Click on the button to Edit the space
4. In the drawer that appears on the right, expand the Sleep Mode section
5. Use the Delete After Inactivity field to specify the Time (in minutes) to wait before putting the space to sleep if there is no more user activity involving this namespace
6. On the very bottom, click on the button to save the changes

Enforce Auto-Delete Timeout For All Space Created By User/Team

1. Go to the Clusters view using the menu on the left
2. Switch to the Space Constraints tab

3. Option A: Hover over the space constraints object that you want to configure auto-delete with and click on the button to Edit an existing space constraints object

   Option B: Click the button to create a new space constraints object

4. In the drawer that appears on the right, expand the Enforce Space Settings section
5. Use the Delete After Inactivity field to specify the Time (in minutes) to wait before deleting the space if there is no more user activity in this namespace
6. On the very bottom, click on the or button to save the changes
7. Switch to the Cluster Access tab
8. Hover over the cluster access of the user or team that you want to enforce auto-delete for and click on the button to Edit the cluster access
9. In the drawer that appears on the right, expand the Restrictions section
10. Use the Enforce Space Constraints field to select the Space Constraint you edited or created in Step 3 above
11. On the very bottom, click on the button to update the cluster access

Test with Impersonation

After following the steps above, all spaces created using the cluster access in step 7 will now enforce this auto-delete behavior. You can test this behavior by impersonating a user that uses this cluster access.

Inactivity Detection

All requests that are made through Loft count as activity in the namespace.

If your kube-context points to Loft's API server as a proxy before the actual connected cluster's API server, every kubectl request will be an activity and reset the inactivity timeout.

Configuration

Sleep mode is generally configured as annotations on the namespace.

• sleepmode.loft.sh/sleep-after: tells Loft to trigger sleep mode automatically after x seconds of inactivity
• sleepmode.loft.sh/delete-after: tells Loft to delete the namespace automatically after x seconds of inactivity
• sleepmode.loft.sh/force: if "true" tells Loft to force sleep this namespace
• sleepmode.loft.sh/force-duration: the amount of seconds after the space starts sleeping after which no automatic wake up should occur (a value of 0 means infinite). This can be useful if there is an application accessing the space that would directly wake up the space again after it started sleeping.
• sleepmode.loft.sh/delete-all-pods: if "true" will delete all running pods within a namespace as soon as it starts sleeping, besides scaling down deployments, replicasets and statefulsets

Loft checks every space with an automatic sleep mode annotation in a fixed interval of 1 minute.

Exclude Resources From Sleep

Loft allows you to specify resources that should not sleep within a space by providing the annotation sleepmode.loft.sh/exclude: 'true' on either a Deployment, StatefulSet, ReplicaSet or Pod. For example the following Deployment would not sleep if the space is sleeping:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
  annotations:
    sleepmode.loft.sh/exclude: 'true'
spec:
  replicas: 2
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
        - name: busybox
          image: busybox
          command:
            - sleep
            - '3600'

Refine Activity Detection

There are multiple annotations you can set on a namespace to change what counts as activity within that namespace:

• sleepmode.loft.sh/last-activity: this is usually set automatically by Loft after an activity was detected within a namespace, however you can also change this value as you like
• sleepmode.loft.sh/ignore-all: if this annotation is set to "true", then all requests will be ignored and not count as activity
• sleepmode.loft.sh/ignore-vclusters: if "true", activity that occurs within a created vcluster within this namespace does not count as activity
• sleepmode.loft.sh/ignore-groups: a comma separated list of user groups that do not count as activity (e.g. loft:team:admins would ignore all users in the team admins or loft:user:admin would ignore the admin user)
• sleepmode.loft.sh/ignore-resources: a comma separated list of kubernetes resources that do not count as activity (e.g. pods, deployments.apps, secrets etc.)
• sleepmode.loft.sh/ignore-verbs: a comma separated list of kubernetes verbs that do not count as activity (e.g. get, list, create, update, patch, delete)
• sleepmode.loft.sh/ignore-resource-verbs: a comma separated list of resources and verbs that do not count as activity (format: myresource.mygroup=create update delete,myresource2.mygroup=create update)
• sleepmode.loft.sh/ignore-resource-names: a comma separated list of resources and names that do not count as activity (format: myresource.mygroup=name1 name2)
• sleepmode.loft.sh/ignore-active-conntections: if this annotation is set to "true", then still active kubernetes connections will be ignored for determining activity in a space

If you want to set some of this annotations automatically on space creation, you can add these to the space template in the account settings.

Long-Living Connections

It's possible that certain requests such as kubectl exec or kubectl port-forward keep an active connection open to the space that prevents it from sleeping. This is in most cases wanted since the space shouldn't start sleeping when someone is still clearly using it, however there are cases where the connection might be idle (someone left the laptop open, but is not using it anymore) in which you want to terminate such connections and put the space to sleep.

There are multiple ways to approach this problem of active connections preventing a space from sleeping:

1. You can tell loft to timeout idle streaming connections (such as kubectl exec, kubectl port-forward etc.) after a certain time period with the annotation loft.sh/streaming-connection-idle-timeout: '3600' on a cluster. With this annotation set, loft will close connections automatically that are idle after the given seconds. By default, Loft will not timeout any connections. This will only apply to new opened connections to that cluster and not affect already running connections.

   

   Set annotation on cluster

2. Tell loft to ignore all active connections for determining space activity via the space annotation sleepmode.loft.sh/ignore-active-conntections: 'true'. This will put a space to sleep even though there still might be open connections such as kubectl exec or kubectl port-forward.

3. Configure your kubelets with the flag --streaming-connection-idle-timeout duration(see docs). This behaves essentially as option 1, however this configuration is Loft independent.