@SWITCHengines profile picture
SWITCHengines @SWITCHengines

The Swiss Academic Cloud Service

Joined April 2015
  • Tweets 355
  • Following 8
  • Followers 181
13 Photos and videos
All systems are working as expected. We have formally closed the incident. Sorry for the problems this has caused. We are investigating why the switch started to act up. ^jcf
1
We have been able to fix the root cause (a switch was blackholing internal traffic in our compute and storage clusters). This has been fixed for an hour now and we have seen most services recover. Our engineers are monitoring the situation. ^jcf
1
1
We are experiencing problems with various parts of SWITCHengines that lead to various problems with services running on it. We are assembling a team of engineers now to diagnose and work on it. ^jcf
Data, data everywhere...come help us keep it safe recruitingapp-2563.umantis.c…
1
The SWITCHengines dashboard is currently unavailable. VMs are running normally but it is not possible to make changes to VMs or to provision new projects or VMs via the web. API/command line access is still available. We are investigating. ^AHa
Communication via the message queuing system has been restored. All functions are reporting normal operation in Lausanne again. ^AHa
Control plane operations such as starting/stopping/resizing VMs are currently unavailable in Lausanne. Running VMs are not impacted. Underlying connectivity is verified, and we continue to investigate. ^AHa
1
Performance on control plane in ZH degraded again this afternoon. We have increased the resources for key components which has relieved the performance issues. We are continuing to monitor closely while debugging root cause. ^AHa
Performance stabilised in ZH from 13:00 following troubleshooting. It is now possible to provision or deprovision VMs with reasonable response times. Running VMs were not impacted. ^AHa
Since c 11:30 see degraded performance on SWITCHengines control plane Zürich. Running VMs are not impacted but creating, deleting or changing VMs is impacted and some APIs are not responding. We are investigating and may need to restart some systems to restore service. ^AHa
Supporting new services in 2021 - after a successful pilot we're looking for expressions of interest in Office365 backup with your Data@SWITCH @SWITCH_ch ICT Focus.
Operational improvements 2021 - more OpenStack upgrades, new S3 cluster, ISMS maturity @SWITCH_ch ICT Focus
Our strategic focus areas 2021 - Data@ SWITCH, supporting data management and data science use cases for higher education & research in CH @SWITCH_ch ICT Focus
1
We're part of the basic infrastructure roots in the swissuniversities Open Science vision @SWITCH_ch ICT Focus
BlueBrain Nexus on k8s running on SWITCHengines presented at @SWITCH_ch ICT Focus 2020
1
Connectivity in Lausanne has been restored. Cause was a subset of IPv4 routes being blackholed unexpectedly. ^AHa
SWITCHengines@SWITCHengines
22 Oct 2020
We are experiencing network problems in our LS region. This causes VM IPv4 connections interrupted. We are investigating the problem. ^HR
1
We are experiencing network problems in our LS region. This causes VM IPv4 connections interrupted. We are investigating the problem. ^HR
All assignments of IPv4 addresses have been restored. If you still experience problems, please file a ticket with engines-support@switch.ch
2
We accidentally freed and de-allocated 87 floating IP addresses in our ZH cluster. A number of VMs and loadbalancers have lost their IP addresses. Our engineers are working on restoring the IP addresses to their respective owners. ^jcf
2
Our engineers have been restarting the affected network components and connectivity to the LS cluster has been restored. We are now monitoring the situation. If you are still experiencing problems, please get in touch with us through ‘engines-support@switch.ch’
1
1
Load more