tag:status.redpill-linpro.com,2005:/historyRedpill Linpro Status - Incident History2024-03-28T19:40:34+01:00Redpill Linprotag:status.redpill-linpro.com,2005:Incident/202444602024-03-14T22:44:38+01:002024-03-14T22:44:38+01:00BGP route problem<p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>22:44</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>22:43</var> CET</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>16:06</var> CET</small><br><strong>Monitoring</strong> - BGP route problem in OSL3 triggered route reconfiguration for OpenStack. Solution is recovering, but may take time.</p>tag:status.redpill-linpro.com,2005:Incident/107429852022-08-02T12:29:30+02:002022-08-02T12:29:30+02:00Network partially down in the OSL2 datacentre<p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>12:29</var> CEST</small><br><strong>Resolved</strong> - An unplanned boot of a leaf switch caused unwanted behaviour from a core switch. The leaf switch has been configured out of production.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>12:16</var> CEST</small><br><strong>Investigating</strong> - We are currently investigating</p>tag:status.redpill-linpro.com,2005:Incident/96568192022-03-29T12:43:10+02:002022-03-29T12:57:46+02:00FibreChannel infrastructure down in OSL2<p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>12:43</var> CEST</small><br><strong>Resolved</strong> - An automatic source transfer switch failed transferring power to alternate source during planned outage performed by DC personnel. This caused the FC switches in one rack to fail. The problem was corrected by directing the DC personnel to perform a manual failover on the source transfer switch.</p>tag:status.redpill-linpro.com,2005:Incident/57562192020-12-14T09:23:11+01:002020-12-14T09:23:11+01:00Telephony trouble<p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>09:23</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>23:26</var> CET</small><br><strong>Monitoring</strong> - We are still monitoring the problem</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>23:26</var> CET</small><br><strong>Investigating</strong> - As earlier announced, our telephony provider migrated our oncall and contact numbers to a new server. This did not end well. Please use the number provided in the work-announcement mail (+47 47780293).</p>tag:status.redpill-linpro.com,2005:Incident/49920922020-09-01T13:34:08+02:002020-09-01T13:34:08+02:00Problems with "situla"<p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>13:34</var> CEST</small><br><strong>Resolved</strong> - The object gateways seems to work fine now. We have partly identified root cause for the issues we saw today, but not for the initial issues.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>12:13</var> CEST</small><br><strong>Monitoring</strong> - A upgrade of the object gateways created unexpeced problems. The object storage rig seems to be working now. We will keep on monitoring the situation and look for root cause.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>12:06</var> CEST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>12:06</var> CEST</small><br><strong>Investigating</strong> - We are currently working on fixing the problem</p>tag:status.redpill-linpro.com,2005:Incident/43959452020-06-18T07:23:33+02:002020-06-18T07:23:33+02:00Ceph outage<p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>07:23</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>07:16</var> CEST</small><br><strong>Identified</strong> - We are currently experiencing an outage impacting our Ceph storage cluster. By extension, this impacts the Situla object storage service. The root cause has been identified, and we are working on restoring service.</p>tag:status.redpill-linpro.com,2005:Incident/40548782020-05-07T15:58:48+02:002020-05-07T15:58:48+02:00High IO usage on Ceph cluster<p><small>May <var data-var='date'> 7</var>, <var data-var='time'>15:58</var> CEST</small><br><strong>Resolved</strong> - Problem was probably initiated by adding more storage nodes(disks) to the cluster. This creates an extra IO load as the cluster seeks to rebalance the data over the entire diskset including the new and empty disks. The software handling the nodes started sporadically dying when the io load was particularily high, creating further IO loads when restarting.<br /><br />The underlying problem is still there, but the cluster has been reported as more responsive again. Backfills will continue through the next week or two. We will create a new incident on further problems.</p><p><small>May <var data-var='date'> 7</var>, <var data-var='time'>13:35</var> CEST</small><br><strong>Investigating</strong> - We are currently investigating high IO utilization on our Ceph cluster. This will manifest itself as slower response from the filesystem.</p>tag:status.redpill-linpro.com,2005:Incident/40003072020-04-30T21:03:41+02:002020-04-30T21:03:41+02:00Intermittent IO problems with a SAN in OSL2<p><small>Apr <var data-var='date'>30</var>, <var data-var='time'>21:03</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>30</var>, <var data-var='time'>13:39</var> CEST</small><br><strong>Identified</strong> - We are still seeing intermittent IO problems with a SAN in OSL2. We are working on removing more workload.</p>tag:status.redpill-linpro.com,2005:Incident/40002602020-04-28T09:00:00+02:002020-04-30T13:31:55+02:00Intermittent IO problems with a SAN in OSL2<p><small>Apr <var data-var='date'>28</var>, <var data-var='time'>09:00</var> CEST</small><br><strong>Resolved</strong> - We saw intermittent IO problems with a unity SAN in OSL2. We removed some workload from the SAN.</p>tag:status.redpill-linpro.com,2005:Incident/34753162020-02-11T04:46:36+01:002020-02-11T04:46:36+01:00Storage maintenance<p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>04:46</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>01:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>11:17</var> CET</small><br><strong>Scheduled</strong> - Update: This maintenance window has been postponed 1 week.<br /><br />In this maintenance window, we will reconfigure parts of the storage cluster underlying 'ceph-sata' in availability zone 'OSL2', and 'situla' - our S3 compatible storage solution.<br /><br />This will result in increased response times during the maintenance window, but as the maintenance will remove a bottleneck - should also result in better performance in some cases.</p>tag:status.redpill-linpro.com,2005:Incident/34748362020-02-11T04:45:57+01:002020-02-11T04:45:57+01:00Network maintenance in OSL2<p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>04:45</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>01:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>16:40</var> CET</small><br><strong>Update</strong> - Update: this maintenance has been postponed by another week. New time: 2020-02-11 0100-0600 CET.</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>08:54</var> CET</small><br><strong>Scheduled</strong> - In this maintenance window, we intend to upgrade the operating system of some our data centre switches in OSL2 to the latest bug-fix release recommended by the manufacturer.<br /><br />This maintenance only applies to some of rack switches that were originally scheduled for being upgraded 2012-01-28. Due to a customer request, we have decided to partially postpone this maintenance by one week.<br /><br />While the work is in progress, the redundancy in the data centre network will be reduced from 1+1 to 1+0.<br /><br />If the upgrade goes according to plan and no unexpected faults occur while the redundancy is reduced, customers are not expected to experience any significant outage.</p>tag:status.redpill-linpro.com,2005:Incident/35266012020-02-05T23:04:04+01:002020-02-05T23:04:04+01:00Hypervisor outage in OSL3<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>23:04</var> CET</small><br><strong>Resolved</strong> - All impacted instances have been (or will soon be) evacuated away from the problematic hypervisor.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>19:42</var> CET</small><br><strong>Identified</strong> - We have an hypervisor reporting hardware errors in OSL3, impacting some customer VMs. We are working to resolve this issue.</p>tag:status.redpill-linpro.com,2005:Incident/34577382020-01-30T16:00:30+01:002020-01-30T16:00:30+01:00Electricity infrastructure service in OSL2<p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>16:00</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>08:01</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>15:17</var> CET</small><br><strong>Scheduled</strong> - Our data centre operator GlobalConnect has notified us of their intention to perform service on several components of the the electricity infrastructure in our OSL2 data centre.<br /><br />Technical personnel from GlobalConnect and their service partner Coromatic will be on-site while the maintenance is in progress.<br /><br />The maintenance schedule is as follows (all times CET):<br /><br />* 08:00-12:00: Service on the main UPS switch. In this period, the data centre will be powered by the emergency generator. The health of the generator will be continously monitored. Should the generator fail, the data centre will lose power until either the service is complete or the supply is manually changed back to mains supply.<br />* 12:00-16:00: Service on generator 1 supply switch. In this period, the data centre will be powered by the mains supply. Should the mains supply fail, the supply will manually be changed to the generator before the UPS battery banks run empty.</p>tag:status.redpill-linpro.com,2005:Incident/34911752020-01-30T10:48:38+01:002020-01-30T10:48:38+01:00RLNC hypervisors down in OSL3 (gp2.* flavours)<p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>10:48</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>10:43</var> CET</small><br><strong>Update</strong> - Two of three problematic hypervisors are back on line. The cause appears to have been incorrect network cabling. We are working on the last one, which appears to have lost required BIOS settings for virtualisation following planned maintenance.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>10:29</var> CET</small><br><strong>Identified</strong> - We are observing several compute hypervisors hosting gp2.* instance flavours being down in OSL3. We are working to bring them and their instances back online.</p>tag:status.redpill-linpro.com,2005:Incident/34577362020-01-29T16:00:30+01:002020-01-29T16:00:30+01:00Electricity infrastructure service in OSL2<p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>16:00</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>08:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>17:50</var> CET</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>15:16</var> CET</small><br><strong>Scheduled</strong> - Our data centre operator GlobalConnect has notified us of their intention to perform service on several components of the the electricity infrastructure in our OSL2 data centre.<br /><br />Technical personnel from GlobalConnect and their service partner Coromatic will be on-site while the maintenance is in progress.<br /><br />The maintenance schedule is as follows (all times CET):<br /><br />* 08:00-11:00: Service on UPS 1. In this period, UPS 2 and 3 will remain on-line. Fault tolerance will be reduced to 2N (normally 3N).<br />* 11:00-13:30: Service on UPS 2. In this period, UPS 1 and 3 will remain on-line. Fault tolerance will be reduced to 2N (normally 3N).<br />* 13:30-16:00: Service on UPS 3. In this period, UPS 1 and 2 will remain on-line. Fault tolerance will be reduced to 2N (normally 3N).</p>tag:status.redpill-linpro.com,2005:Incident/34505352020-01-28T05:46:38+01:002020-01-28T05:46:38+01:00Network maintenance in OSL2<p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>05:46</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>01:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>18:36</var> CET</small><br><strong>Scheduled</strong> - In this maintenance window, we intend to upgrade the operating system of our data centre switches in OSL2 to the latest bug-fix release recommended by the manufacturer.<br /><br />While the work is in progress, the redundancy in the data centre network will be reduced from 1+1 to 1+0.<br /><br />If the upgrade goes according to plan and no unexpected faults occur while the redundancy is reduced, customers are not expected to experience any significant outage.<br /><br />Note: some of our customers have private leased circuits connected to the equipment that will be upgraded. Some of those circuits do not have any redundancy, and will as a result experience a longer downtime. The customers in question will be notified directly.</p>tag:status.redpill-linpro.com,2005:Incident/34577302020-01-27T16:00:34+01:002020-01-27T17:51:50+01:00Electricity infrastructure service in OSL2<p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>16:00</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>08:01</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>15:15</var> CET</small><br><strong>Scheduled</strong> - Our data centre operator GlobalConnect has notified us of their intention to perform service on several components of the the electricity infrastructure in our OSL2 data centre.<br /><br />Technical personnel from GlobalConnect and their service partner Coromatic will be on-site while the maintenance is in progress.<br /><br />The maintenance schedule is as follows (all times CET):<br /><br />* 08:00-12:00: Service on the mains supply switch. In this period, the data centre will be powered by the emergency generator. The health of the generator will be continously monitored. Should the generator fail, the power supply will be manually changed back to the mains supply before the UPS battery banks run empty.<br />* 12:00-14:00: Service on the generator supply switch. In this period, the data centre will be powered by the mains supply. Should the mains supply fail, the supply will manually be changed to the generator before the UPS battery banks run empty.<br />* 14:00-16:00: Service on the UPS bypass switch. In this period, the data centre will be powered normally and have normal redundancy. (The UPS bypass switch is only used while servicing UPS-es.)</p>tag:status.redpill-linpro.com,2005:Incident/34222182020-01-21T04:17:47+01:002020-01-21T04:17:47+01:00Network maintenance in OSL3<p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>04:17</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>01:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>14:34</var> CET</small><br><strong>Scheduled</strong> - In this maintenance window, we intend to upgrade the operating system of our data centre switches in OSL3 to the latest bug-fix release recommended by the manufacturer.<br /><br />While the work is in progress, the redundancy in the data centre network will be reduced from 1+1 to 1+0.<br /><br />If the upgrade goes according to plan and no unexpected faults occur while the redundancy is reduced, customers are not expected to experience any significant outage.<br /><br />Note: some of our customers have private leased circuits connected to the equipment that will be upgraded. Some of those circuits do not have any redundancy, and will as a result experience a longer downtime. The customers in question will be notified directly.</p>tag:status.redpill-linpro.com,2005:Incident/34214002020-01-15T08:25:10+01:002020-01-15T08:25:11+01:00Fibre cut in Økern area<p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>08:25</var> CET</small><br><strong>Resolved</strong> - Global Connect has now repaired the damaged fibre cable, all impacted circuits came back up at 08:10 local time.</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>09:59</var> CET</small><br><strong>Update</strong> - Global Connect is confirming that there is an outage in their optical fibre network impacting multiple customers of theirs.<br /><br />The reference number is INC000000419337. Updates from Global Connect may be tracked at https://www.globalconnect.no/driftsmeldinger (Norwegian).</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>09:51</var> CET</small><br><strong>Identified</strong> - Our monitoring systems are reporting multiple link down events in our optical fibre ring in Oslo. The cut appears to be in the Økern area, and occurred at 09:05 local time. The fault has been escalated to our optical fibre provider Global Connect.<br /><br />There are no observations of the fibre cut causing any impact to customer systems at this time, as traffic is automatically re-routed across the ring in the opposite direction. However, the network currently operates with reduced fault tolerance and capacity.</p>tag:status.redpill-linpro.com,2005:Incident/33949622020-01-14T06:00:30+01:002020-01-14T06:00:30+01:00Network maintenance in OSL4<p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>06:00</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>01:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>12:54</var> CET</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>12:53</var> CET</small><br><strong>Update</strong> - Update incorrect specification of components affected.</p><p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>12:51</var> CET</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>12:51</var> CET</small><br><strong>Scheduled</strong> - In this maintenance window, we intend to upgrade the operating system of our data centre switches in OSL4 to the latest bug-fix release recommended by the manufacturer.<br /><br />While the work is in progress, the redundancy in the data centre network will be reduced from 1+1 to 1+0.<br /><br />If the upgrade goes according to plan and no unexpected faults occur while the redundancy is reduced, customers are not expected to experience any significant outage.</p>tag:status.redpill-linpro.com,2005:Incident/34238322020-01-13T23:10:28+01:002020-01-13T23:10:28+01:00Impaired Internet connectivity due to DDoS attack<p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>23:10</var> CET</small><br><strong>Resolved</strong> - The attack appears to have ceased completely at around 22:57 local time.<br /><br />It was a DNS amplification DDoS attack directed at a specific customer of ours.</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>22:49</var> CET</small><br><strong>Monitoring</strong> - The attack began at 22:12 local time, and was large enough to congest several of our IP transit links. This will have caused degraded performance and packet loss on traffic from/to the Internet.<br /><br />Our two IP transit providers started mitigation (scrubbing) at 22:17 and 22:35.<br /><br />The attack is still ongoing, but the small amount of attack traffic that still makes it through to our network no longer poses any problem for us. We will continue to monitor the situation.</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>22:40</var> CET</small><br><strong>Identified</strong> - We have been hit with a DDOS attack and are mitigating.</p>tag:status.redpill-linpro.com,2005:Incident/31354212019-10-23T17:02:50+02:002019-10-23T17:02:50+02:00Unexpected hypervisor reboots in OSL3<p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>17:02</var> CEST</small><br><strong>Resolved</strong> - All impacted virtual machines are back up.</p><p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>16:59</var> CEST</small><br><strong>Identified</strong> - The hypervisors are back up, and the disrupted VMs are being started back up again.</p><p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>16:48</var> CEST</small><br><strong>Investigating</strong> - In conjunction with work in the data centre, two hypervisors in the OSL3 data centre unexpectedly went down. We are working on it.</p>tag:status.redpill-linpro.com,2005:Incident/30453352019-10-11T02:00:30+02:002019-10-11T02:00:30+02:00Bahnhof IP transit maintenance<p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>02:00</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>00:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>09:26</var> CEST</small><br><strong>Scheduled</strong> - One of our two IP transit providers, Bahnhof, have advised us of their intention to perform maintenance work. They anticipate the service to be impacted for a total of about 30 minutes in the specified maintenance window.<br /><br />As Redpill Linpro also receive IP transit from another provider, Telenor, we do not anticipate this maintenance to cause any significant outage for our customers. The BGP routing protocol used should ensure that traffic that normally pass through Bahnhof will be automatically re-routed onto alternate paths, either within Bahnhof's core network, or by avoiding Bahnhof altogether by moving to Telenor.<br /><br />These re-routing events could cause brief disturbances as the network converges onto the alternate paths, however.</p>tag:status.redpill-linpro.com,2005:Incident/29792742019-09-30T06:00:29+02:002019-09-30T06:00:29+02:00Dark fibre maintenance<p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>06:00</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>00:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>15:02</var> CEST</small><br><strong>Scheduled</strong> - Our dark fibre provider has advised us of their intent to perform maintenance of one of our fibre segments in Oslo. This will involve a temporary disconnection lasting between three and six hours.<br /><br />The dark fibre circuits that connect our three data centres are arranged in a geographically diverse ring structure. Traffic that normally flows through the impacted segment will automatically be re-routed to traverse the ring in the opposite direction. Therefore, we do not anticipate this work to cause any noticeable impact for our customers.<br /><br />Our connection to the Norwegian Internet Exchange goes through the impacted fibre segment. This traffic will be automatically re-routed to one of our IP transit providers instead.</p>tag:status.redpill-linpro.com,2005:Incident/29572892019-09-19T06:33:24+02:002019-09-19T06:33:24+02:00Re-routing of dark fibre circuit in Oslo<p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>06:33</var> CEST</small><br><strong>Completed</strong> - This maintenance was cancelled. Our fibre provider discovered that it was not necessary after all.</p><p><small>Sep <var data-var='date'>17</var>, <var data-var='time'>09:11</var> CEST</small><br><strong>Update</strong> - Our dark fibre provider postponed this work until tomorrow due to a personnel shortage.</p><p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>11:43</var> CEST</small><br><strong>Scheduled</strong> - Our dark fibre provider has advised us of their intent to re-route one of our fibre segments in Oslo.<br /><br />In order to do so, they need to disconnect the current segment and reconnect it to the new route. This is expected to take less than 10 minutes.<br /><br />The dark fibre circuits that connect our three data centres are arranged in a geographically diverse ring structure. Traffic that normally flows through the impacted segment will automatically be re-routed to traverse the ring in the opposite direction. Therefore, we do not anticipate this work to cause any noticeable impact for our customers.</p>