20:45
We have been monitoring services over the weekend and today and so far the network has remained stable.
The supplier remains cautious and will be continuing to monitor the network for any further issues for at least the next few days.
We are aware of a small number of websites that for certain customers are not accessible and are working with those customers and the supplier’s NOC team directly to resolve.
If your service was affected and has not been restored as of yet please reboot any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels
16:45
We have received the below update from the supplier:
A considerable amount of changes and work has been completed during the day today including additional upstream capacity and changes to the filtering to protect our network. There is ongoing changes and updates ongoing this evening to bring the network stability back to what we would expect to provide our customers with.
Our technical team have also been in touch with the supplier directly and understanding more about the work still ongoing believe issues should be largely resolved this evening assuming all goes well.
We will be monitoring this issue this evening and will provide further updates as they become available.
Once again we apologise for the ongoing issues.
13:30
To the follow-up from the drop this morning, this is currently being worked on by the supplier’s senior team, the up-stream NOC team & the transit providers to ascertain what occurred and resolve fully.
In the meantime to get things back to a working state for most customers they have had to roll back some of the changes they’d previously made which has effected some of the mitigations for SIP and VPN traffic. They are working on restoring these as quickly as possible.
We are aware some customers still have lingering issues outside of SIP/VPN traffic and these are being looked into on a case by case basis.
We will update further when we know more and apologise for the ongoing issues.
09:45
We are aware another drop on services has just occurred.
We have reported this to the supplier and are waiting for an update as to what has caused it. We will update further when we know more.
Once again, we apologise for these ongoing issues.
08:00
The last update we received on this from the supplier last night indicated work was still ongoing to re-route affected traffic with the upstream transit provider overnight and to expect a further update this morning.
As soon as we have received this we will update further. Please note there may be some delay between us receiving an update from a supplier and being able to pass it on to customers.
18:45
Please see most recent update from supplier below on this issue:
Good evening. We are are working with the senior teams at the up-stream provider who have moved a large amount of the traffic that was being affected away from the routes. We are seeing improvements in the situation but we are also aware that some of our customers are still impacted.
We will update further when we know more. Please note there may be some delay between us receiving an update from a supplier and being able to pass it on to customers.
16:30
We are aware of some services still experiencing lingering issues and the supplier has confirmed they are related to this outage. They are working on trying to resolve these remaining issues and have provided the following update:
We have seen some improvement to services since our last update but just to confirm that we are still working on mitigation of the transit problems with the upstream provider; once we have a better view on timings etc we will add a further update here and in any event, one at 6PM.
We will update further when we know more. Please note there may be some delay between us receiving an update from a supplier and being able to pass it on to customers.
15:15
We are aware of some services still experiencing lingering issues and the supplier has confirmed they are related to this outage. They are working on trying to resolve these remaining issues as soon as possible.
We will update further when we know more.
14:00
Within about 45 minutes of issues reoccurring we saw them settle back down again and services begin to return to normal, we are currently monitoring whilst we await a further update from the supplier.
We will update further when we know more.
12:30
Unfortunately we have seen a re-occurrence of this issue on our monitoring and the supplier has confirmed it appears the same upstream transit provider is once again having issues:
Our apologies; one of our up-stream transit providers is warning of route instability again. We were told this had been resolved but are actively working with them to understand why this had recurred and mitigate the cause. Our sincere apologies.
We will update further when we know more.
20:30
We have received a further update on this issue from our suppliers who have confirmed this should now be resolved. They are seeing traffic flowing normally again and are not seeing any lingering issues. From our own limited testing we also are not seeing any lingering issues.
As mentioned previously this issue was caused by a fault with an upstream transit supplier which affected multiple services.
If your service was affected and has not been restored as of yet please reboot any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
16:30
We have received a further update on this issue and work is still ongoing to fully resolve and re-route affected traffic. Most services have come back online with minimal lingering issues and those that are still affected are largely seeing only intermittent packet loss. Currently whilst a firm ETA has not been given, we have been told to expect issues to be fully resolved sometime this evening and will be continuing to monitor the situation.
Issues may manifest as intermittent or permanent loss of service. In the case of Phone services this could also manifest as poor call quality.
We will update you further when we know more.
15:00
We have received a further update on this issue and now know the issue is with an upstream transit provider affecting multiple services. Suppliers are working to mitigate and route around. This is why the issue appears to be intermittent and for some customers is no longer affecting them.
Issues may manifest as intermittent or permanent loss of service. In the case of Phone services this could also manifest as poor call quality.
We will update you further when we know more.
14:30
We are aware of an outage affecting various internet based services since about 14:00.
The supplier is aware of the issue and engineers are engaged to work on a resolution.
Below is a list of some of the services that may be affected:
Remote Desktop/MyDesktop Services.
Hosted Phones.
Broadband.
Web Hosting.
Issues may manifest as intermittent or permanent loss of service. In the case of Phone services this could also manifest as poor call quality.
We will update you further when we know more.
Please be aware of upcoming maintenance that affects your service:
Service: CityFibre
Customer Impact: Service Affecting
Location: Peterborough
Reason for Maintenance: To resolve ongoing network issues supplier has been experiencing in Peterborough following guidance from the hardware vendor.
Description of Works: To upgrade and replace Core Router PB1244 in Peterborough.
Start Date and Time of Maintenance: 09/06/21 01:00
End Date and Time of Maintenance: 09/06/21 06:00
Time of Maintenance: 5 Hours.
Additional Info: Customers will experience an up to 5 hour outage during the maintenance window.
If your service is affected and is not restored following the completion of the maintenance please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
Please be aware of upcoming maintenance that affects your service:
Service: CityFibre
Customer Impact: Service Affecting
Location: Peterborough
Reason for Maintenance: This is required due to an integrated port not being operational.
Description of Works: To replace Core Router PB1244 in Peterborough.
Start Date and Time of Maintenance: 10/05/21 01:00
End Date and Time of Maintenance: 10/05/21: 05:00
Time of Maintenance: 4 Hours.
Additional Info: Customers will experience an up to 4-hour outage during the maintenance window.
If your service is affected and is not restored following the completion of the maintenance please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
23:30
We have received a further update from the supplier who now believes this issue fully resolved, please see below:
Once again, please accept our apologies for the problems you’ve seen over the previous couple of days. We realise that this has caused you all serious issues and for that, we’re very sorry.
Various internal changes have been implemented over the last 48 hours and currently, we believe that the network and associated services are now stable and will remain that way. We continue to monitor the situation closely to ensure that our network remains stable and there’s no further impact to your services.
Currently from what we can see services appear to have returned to normal and look to be stable and have been for some time now. This issue as previously mentioned was due to a large scale DDoS (Distributed Denial of Service) attack against one of our supplier’s networks.
If your service was affected and has not been restored as of yet please reboot any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
16:30
We are once again seeing services start to settle down again and stabilise.
We are continuing to monitor the situation.
We will update here further when we know more.
15:30
We are aware that after a period of relative stability over the last few hours another drop has occurred.
The supplier is aware and working to resolve.
Unfortunately the DDoS attack that is the cause of these issues has not stopped in its intensity since it began yesterday and as a result the increased load on the network is causing intermittent drops and issues.
We will update you as soon as we know more.
10:00
We are aware service has gone down again and the supplier is investigating:
We are aware of issues still ongoing; we are actively working on these as before to mitigate their effects. We don’t have an ETA at the moment but we will update as soon was we do.
We will provide further updates as soon as we know more.
09:00
The supplier continued to apply mitigations works overnight and are monitoring the situation today. So far this morning we are seeing services remain stable.
A further update is expected by midday.
We will update further when we have received this.
16:45
We have received a further update from the supplier on this issue.
The DDoS attack itself is still occurring but the supplier is continuing to implement remediation works to reroute traffic and alleviate the load on the network. As such we have seen a number of services come back online however issues are still expected until the DDoS is fully mitigated.
We will continue to provide updates as we know more.
15:00
We are aware this issue remains ongoing and the supplier is working on resolving.
We apologise for the delayed update, the outage is affecting our access to certain services and systems and we are working on putting workarounds in place where possible to keep everyone up to date.
Our support desk ticketing system remains largely unaffected at this time but there is a backlog of requests from the outage slowing our responses there and we are doing our best to clear this backlog and update everyone.
We will update when we know more.
14:00
The supplier has advised they now believe this issue resolved and we are seeing services return to normal for affected customers.
Please see below update from the supplier:
We are seeing most services recovering. The attack target remains offline but we believe that this incident is now contained. We apologise again for this interruption in service. If you are still seeing issues, please restart your equipment.
This issue as previously mentioned was due to a large scale DDoS (Distributed Denial of Service) attack against one of our core supplier’s network.
If your service was affected and the issue is not resolved please get in touch with our support team via the usual channels.
13:35
We are aware of an outage affecting various internet based services. This is due to a large scale DDoS (Distributed Denial of Service) attack against one of our core supplier’s network.
The supplier is aware of the issue and has provided the following update:
We are mitigating a large portion of this attack traffic but currently, the transit links remain saturated which is causing the current ongoing problems. We continue to work to resolve this as quickly as possible & apologise for the ongoing inconvenience caused.
We will update you further when we know more.
13:00
We are aware of an outage affecting various internet based services. This is due to a large scale DDoS (Distributed Denial of Service) attack against one of our core supplier’s network.
The supplier is aware of the issue and engineers are engaged to work on a resolution.
Below is a list of some of the services that may be affected:
Remote Desktop/MyDesktop Services.
Hosted Phones.
Broadband.
Web Hosting.
Issues may manifest as intermittent or permanent loss of service. In the case of Phone services this could also manifest as poor call quality.
We will update you further when we know more.
10:30
We have identified the cause of the issue and put a fix in place. We are now seeing users able to connect without issue.
As such we believe this issue to now be resolved. If you are still experiencing issues please contact support via the usual channels.
09:30
We are aware of an issue preventing some MyDesktop customers from logging on this morning.
Engineers are investigating the cause of this issue and we will update here when we know more.
CityFibre have advised this maintenance did not go ahead as planned due to unforeseen circumstances and has now been rescheduled as per below:
Service: CityFibre
Customer Impact: Service Affecting
Location: Peterborough
Reason for Maintenance: Due to ongoing issue in Peterborough, maintenance is required to bring stability to the network.
Description of Works: Reboot of Peterborough PB1244 Core Router.
Start Date and Time of Maintenance: 26/03/21 00:01
End Date and Time of Maintenance: 26/03/21: 03:00
Time of Maintenance: 3 Hours.
Additional Info: Customers will experience approx. 30 minute outage during the maintenance window.
If your service is affected and is not restored following the completion of the maintenance please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
Please be aware of upcoming maintenance that affects your service:
Service: CityFibre
Customer Impact: Service Affecting
Location: Peterborough
Reason for Maintenance: Due to ongoing issue in Peterborough, maintenance is required to bring stability to the network.
Description of Works: Reboot of Peterborough PB1244 Core Router.
Start Date and Time of Maintenance: 25/03/21 00:01
End Date and Time of Maintenance: 25/03/21: 03:00
Time of Maintenance: 3 Hours.
Additional Info: Customers will experience approx. 30 minute outage during the maintenance window.
If your service is affected and is not restored following the completion of the maintenance please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
23:00
CityFibre have advised the fault has been fixed and this issue should now be fully resolved:
We have had confirmation that this issue is now resolved and that services are no longer at risk. Following assessments by our engineering resource at multiple locations, the damage previously mentioned was unrelated and not the cause of the issue on the remaining impacted service. Instead, a fibre microbend was identified and corrected, with testing demonstrating expected results.
If your service was affected and has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
17:15
We are aware of an issue affecting some CityFibre circuits across the UK causing either intermittent service drops or total loss of service.
CityFibre are aware of this issue and have provided the following update:
After further optimisation of the network we are seeing restoration to the affected customers.
As the there is still an open incident on the Leeds to Sheffield link, services should be considered at risk until the link has been brought back into service.
If your service was affected and has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance.
16:15
We are aware of an issue affecting some CityFibre circuits across the UK causing either intermittent service drops or total loss of service.
CityFibre are aware of this issue and have provided the following update:
After the initial re-routing we are still seeing a portion of customers impacted by the major incident. Further work is being carried out in order to optimise network traffic and alleviate the issues being experienced. We also continue to press our third party supplier for updates on the failure on the link between Leeds and Sheffield. Further update will follow at 17:00
We will update here as we know more.
15:45
We are aware of an issue affecting some CityFibre circuits across the UK causing either intermittent service drops or total loss of service.
CityFibre are aware of this issue and have provided the following update:
The incident is related to an issue on a link between Leeds and Sheffield. Third party providers have already been engaged and are investigating. In the meantime, to mitigate the impact we have re-routed traffic over an alternative link. We are now in the process of assessing the impact of the re-routing. A further update will follow at 16:00.
We will update here as we know more.
15:15
We are aware of an issue affecting some CityFibre circuits across the UK causing either intermittent service drops or total loss of service.
CityFibre are aware of this issue and have engineers investigating. Initial diagnostics have confirmed this is not specific to any one geographic location.
We will update here as we know more.
14:30
We are aware of a brief issue that affected some services hosted in our Datacentre.
This appears to have been a routing issue with one of the links into the Datacentre and is already resolved.
We are monitoring for any further issues.
14:30
We have discussed this issue further with CityFibre and received clarity on the workaround that has been put in place.
After extensive testing they believe the issue appears to be with some of the Fibre ports on one of the Core Routers itself and so they have moved the effected links to alternative known good ports on the router. As such there should be no noticeable impact to service or reliability going forward from this.
CityFibre will continue to investigate the problematic ports and will look to reintroduce these into production at a later date if/when the issues with them have been resolved and extensively tested prior to doing so. This will be part of planned scheduled maintenance at a later date.
As such this issue is considered resolved.
07:45
CityFibre have managed to restore service and have provided the following update:
Engineers have restored customer services although this is only a temporary solution at the moment as a workaround is in place. Further investigation work will be undertaken and a permanent solution will be introduced through our change management process. We will update when this will happen. Apologies for the issue and the disruption this will have caused.
We have requested from CityFibre more information on the temporary solution put in place and if customers should expect further issues whilst the permanent solution is being worked on or not and we will update here if we hear back.
We can see many of the services that we monitor are now back online and have been since the early hours of this morning. Failover circuits where present have failed back across successfully.
If your service was affected and has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
00:15
CityFibre have confirmed there is a small number of circuits with issues in Peterborough today resulting in a total loss of service. They believe this to be a Fibre issue and Engineers are on site:
Engineer parts have arrived on site and further work is ongoing, alongside our NOC team.
Further updates will be provided, once we have any new information from the engineers.
We will update here when we know more
21:45
CityFibre have confirmed there is a small number of circuits with issues in Peterborough today resulting in a total loss of service. They believe this to be a Fibre issue and Engineers are on site:
Replacement optics have been sourced that are compatible with both Cisco and Nokia (MSA compatible). The link from PB1244 – PB1344 is 12km, these optics are 80km so have requested attenuators to align the link up with the required light level.
Engineer resources have been arranged to collect parts and attend both sites.
Further updates will follow once parts and engineer resources have arrived on-site.
We will update here when we know more.
20:10
CityFibre have confirmed there is a small number of circuits with issues in Peterborough today resulting in a total loss of service. They believe this to be a Fibre issue and Engineers are on site:
Engineer resources have confirmed no lights within the fibre is being received between PB1344 and PB1244. They have also proved that the port is up which suggests the SFP is not related to the issue and lies within the fibre.
Further updates will follow once the engineer resources have reached PB1244.
We will update here when we know more.
19:10
CityFibre have confirmed there is a small number of circuits with issues in Peterborough today resulting in a total loss of service. This appears to be the same customers that were affected by a previous fault. They believe this to be a Fibre issue and Fibre Engineers are on site:
Engineer resources are on-site who continue to investigate the issue. We currently have fibre engineers and hardware teams present working alongside our NOC team.
We will provide a further updates in the next hour.
We will update here when we know more.
17:50
CityFibre have confirmed there is a small number of circuits with issues in Peterborough today resulting in a total loss of service. This appears to be the same customers that were affected by a previous fault. They believe this to be a Fibre issue and Fibre Engineers are now on site:
Engineer resources have now arrived on-site where investigations have commenced. We will provide further updates in the next hour.
We will update here when we know more.
16:45
CityFibre have confirmed there is a small number of circuits with issues in Peterborough today resulting in a total loss of service. This appears to be the same customers that were affected by a previous fault. They believe this to be a Fibre issue and are arranging Fibre engineers as per below:
Our NOC team have continued to investigate and believe the issue is fibre related rather than hardware. They have spoken with our ERS contractor in advance of them attending site and to ensure that they are aware of the actions needed. We have also engaged with our first line maintenance teams in case anything additional is needed and have them on standby. As soon as we are able to confirm planned arrival on site, we will update.
We will update here when we know more.
14:45
CityFibre have confirmed there is a small number of circuits with issues in Peterborough today resulting in a total loss of service. This appears to be the same customers that were affected by a previous fault. Engineers are on-route to Peterborough as per below:
Our Team have advised that it is the same link at fault as earlier in the week. However, the fault conditions are different, so it does not appear to be a repeat fault. We have undertaken several steps to clear any issues with our hardware and cards, and we can see that other ports etc. are up and running. We’ve interrogated the logs and events and can see nothing untoward other than a loss of light from one side. Therefore, we are arranging engineers to attend the site and will update with and ETA as soon as we are able.
We will update here when we know more.
14:05
We have had alerts on our monitoring of several CityFibre circuits going offline in Peterborough in the last 15 minutes.
This issue has been raised with CityFibre to investigate further.
We will update here when we know more.
07:35
CityFibre have advised work to replace the faulty card was completed overnight and that this issue should now be resolved:
03:25: Engineer is on site and about to commence work. Update will follow upon completion of the task.
04:55: The card has been replaced, and we have collected further diagnostic information. We will confirm operational status once this has been reviewed by our vendor.
05:25: We believe this issue to now be resolved, and request that any affected users retest their service. Please accept our apologies for any inconvenience that may have been caused.
We can see many of the services that we monitor are now back online and have been since the card swap took place. Failover circuits where present have failed back across successfully.
If your service was affected and has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
00:15
CityFibre have advised that the maintenance has completed successfully however it has highlighted another potential faulty component that is still causing issues for some circuits and they will continue to work on this overnight:
Our vendors are dispatching an engineer to replace a suspected faulty card in our PB1344-rtr-1 node. ETA is 02:30.
All circuits that were online prior to the maintenance we believe should now be back online. If this is not the case please reboot your CityFibre ONT and any related CPE in the first instance. Circuits that were down prior to the maintenance will still be affected by the above issue that is being worked on.
We will provide a further update here in the morning.
16:45
CityFibre have issued a notice that they will be performing emergency maintenance this evening related to this issue:
Tuesday 23rd February 2021, 19:00 – Tuesday 23rd February 2021, 23:59
During the above window, we will be completing emergency maintenance to upgrade and replace our core router at the above location. This is required in order to resolve the issues we have been experiencing in Peterborough. There will be a small window of downtime as we replace the chassis and migrate services across.
This will affect ALL CityFibre services in Peterborough.
16:05
CityFibre have confirmed there are a small number of circuits with ongoing issues today and that they are still investigating:
Engineer resources have now arrived on-site where investigations have commenced. We will provide further updates in the next hour.
We will update here when we know more.
14:15
CityFibre have confirmed there are a small number of circuits with ongoing issues today and that they are still investigating:
Based on NOC troubleshooting we have engaged engineer resources to attend the affected site and carry out further testing.
Engineer resources have been arranged with an ETA of 16:00. We will provide a further update upon the engineers arrival.
We will update here when we know more.
12:25
CityFibre have confirmed there are a small number of circuits with ongoing issues today and that they are still investigating:
We are still investigating this issue with our senior agents within the NOC team and will provide further updates in the next hour.
We will update here when we know more.
10:45
CityFibre have confirmed there are a small number of circuits still with lingering issues this morning and that they are investigating:
Following the outage last night affecting Peterborough, we are still aware of a small number of circuits still down. Our NOC team are currently investigating the situation and we will provide a further update in the next hour.
We will update here when we know more.
23:55
CityFibre have confirmed all circuits should be back online and this issue should now be resolved.
We have requested a full RFO (Reason for Outage) Report for this issue and will make this available to our customers upon request when we have received this.
If your service was affected and has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
23:15
CityFibre have advised they believe all circuits should be back online however we are still seeing several circuits that are down and are querying this with them.
We will update here when we know more.
22:45
We have seen on our monitoring a few circuits come back online but not all.
No official update from CityFibre as of yet.
We will update here when we know more.
22:15
CityFibre have advised the additional engineering resource has arrived and is working on the issue with the engineers already on site.
Further updates are expected after 23:00.
We will update here when we know more.
21:45
CityFibre have advised after initial diagnostics by engineers on site further engineering resource was required.
This engineering resource was expected to arrive by 21:35.
We do not as of yet have any further update.
We will update here when we know more.
17.15
We have had the following update from CityFibre:
Engineer resources have been engaged to attend the affected site with an ETA of 19:00. Further updates will be posted upon the engineer’s arrival.
We will update here when we know more.
16:25
We have had alerts on our monitoring of several CityFibre circuits going offline in Peterborough in the last 30 minutes.
This issue has been raised with CityFibre who advised they are aware of the issue and are already investigating.
We will update here when we know more.
18:00
CityFibre have confirmed all circuits should be back online and this issue should now be resolved.
We have requested a full RFO (Reason for Outage) Report for this issue and will make this available to our customers upon request when we have received this.
If your service was affected and has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
17:15
All CityFibre circuits that went down between 16:30 – 17:00 appear to now be back online.
We have chased CityFibre for an explanation and further update on the main fault and will update here when we know more.
As mentioned previously the brief outage just now was not planned so we could not warn anyone ahead of time. We will be querying why this was the case.
If your service was affected and has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
16:30
We are aware some CityFibre circuits in Peterborough that were up have just gone down again.
A CityFibre engineer was due onsite in Peterborough at 16:00 and we believe this is most likely related but we are attempting to confirm with CityFibre as they have not advised us of any expected downtime for working sites.
We will update here when we know more.
14:30
CityFibre have advised they are still seeing issues in certain areas of the UK.
They have provided the following update:
We are seeing the majority of services now restored apart from those directly connected through Peterborough. We are now awaiting an engineer to attend Peterborough sites with an ETA of 16:00, the technical teams will then work with the engineers to restore the services connected to Peterborough sites.
We will update here further as soon as we know more.
13:30
CityFibre have advised they are still seeing issues in certain areas of the UK.
They have provided the following update:
Work has been undertaken to return stability which involved shutting down one of our links. Following this, we have seen some services restored however, customers in the Peterborough area may still be impacted. We are currently waiting for engineer resources to attend Peterborough and carry out further troubleshooting.
Once Peterborough services have been restored we will then look to put a permanent fix in place ensuring any changes goes through the change management process due to potential impact to services. Further updates will be provided at 14:15
We will update here further as soon as we know more.
12:30
CityFibre have advised they are still seeing issues in certain areas of the UK.
They have provided the following update:
Investigations into the issue are ongoing. Following internal technical calls, the resolver teams are currently focusing efforts on stabilising the impacted services. Once these have been stabilised, they can then focus on finding the root cause and a permanent fix. Another internal technical call is taking place at 12:30 and further updates will be provided following this at 13:15
We will update here further as soon as we know more.
11:30
CityFibre have advised they are still seeing issues in certain areas of the UK.
They have provided the following update:
NOC and architecture & engineering teams are currently working with external vendor support to find the root cause and restore services. Further updates will be provided in the next hour.
We will update further as soon as we know more.
10:30
CityFibre have advised they are still seeing issues in certain areas of the UK but currently we are not seeing this affect any of our customers.
They have provided the following update:
We are currently seeing a recurrence of the issues following the work earlier this morning. Senior network engineers are continuing to investigate the root cause and further updates will be provided as soon as possible.
High level investigations into the issue are continuing to take place. At present, the problem will cause packet loss, latency or instances of total loss of service and is affecting multiple tunnels across the CityFibre network.
We will update further as soon as we know more.
07:30
We can see many services for customers in Peterborough appear to be back online and stable this morning. Most services appear to have come back online around 03:00.
CityFibre have provided the following update:
Following a re-boot of a core device on-site, services have begun to restore across the network for Peterborough connections.
We now advise to ensure that any services still down, are fully re-booted, and should the issue persist, please contact the service desk to advise the remaining connection issues.
A full report of the root cause is still currently ongoing and to be provided at a later time.
If your service has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
23:00
CityFibre have confirmed they are aware of an MSO (Major Service Outage) in Peterborough affecting a large number of customers and they have dispatched engineers to investigate further. Further updates to follow.
We will update here when we know more.
22:25
We have had alerts on our monitoring of several CityFibre circuits going offline in Peterborough in the last 30 minutes.
This issue has been raised with CityFibre to investigating further.
We will update here when we know more.
17:15
CityFibre have advised this MSO (Major Service Outage) also covers Sheffield and Nottingham in addition to Peterborough.
We are now seeing good stability of connections that have come back online and have received a further update from CityFibre to advise that this issue should now be resolved.
Services have been restored. A reboot was required which has resulted in the data necessary to carry out further root cause analysis to be lost. A reboot in this instance was the last available option in order for us to restore service.
Due to this we will continue to monitor for the next 48hours.
If service has not been restored as of yet please reboot CityFibre ONT and any related CPE and get in touch if still experiencing issues.
We are not currently expecting further related issues at this time. Please monitor and advise if further issues occur via usual channels.
16:45
CityFibre have advised this MSO (Major Service Outage) also covers Sheffield and Nottingham in addition to Peterborough.
We are starting to see some connections come back online for some customers now but have not received an all clear as of yet from CityFibre or a further update.
Last we heard just prior to this the engineer was attempting to restore connectivity to the affected Core Routers.
Until we hear further from CityFibre connectivity is still considered ‘at risk’ for the time being and further drops may occur.
14:30
CityFibre have advised this MSO (Major Service Outage) also covers Sheffield and Nottingham in addition to Peterborough.
Please see further update below:
Situation:
Services connected to our Peterborough POP seen to be down
Action:
Engineers attending to investigate further
Next Step(s):
Engineers are expected to arrive on site at around 1500 to carry out investigations.
Expectation:
Further update to follow once engineer has attended and assessed the situation.
13:00
CityFibre have advised this MSO (Major Service Outage) also covers Sheffield and Nottingham in addition to Peterborough.
We have received a brief update from them a few minutes ago:
Sheffield and Nottingham core routers are back online but we still have an issue in Peterborough. Engineers are being dispatched.
12:00
We are aware of an outage in Peterborough affecting CityFibre customers.
CityFibre are aware of the issue and are investigating. As soon as we know more we will update here and anyone who has raised a support request with us.
17:00
Microsoft Support now believe this issue is resolved. Please see the full update below:
User Impact: Users may have been unable to access the Exchange Online service through the Outlook client.
More info: Additionally, some users may have also experienced impact connecting through Outlook on the web.
Final status: We’ve determined that a subset of components, responsible for directing user requests, received traffic whilst it was going through an update and was not configured correctly to process the traffic. We’ve routed connections to alternate components and confirmed that the issue is remediated after monitoring the service.
Scope of impact: Customers connecting to the Exchange Online service through our Europe and United Kingdom data centers may have experienced impact.
Preliminary root cause: A small section of front end components, used to direct user requests, was inadvertently sent traffic whilst it was receiving an update and was not correctly configured to process requests, which resulted in impact.
We are seeing service return to normal for our users. If you are still experiencing issues please get in touch through the usual support channels.
15:45
We have received a further update from Microsoft Support on this issue. Please see below:
User Impact: Users may be unable to access the Exchange Online service through the Outlook client.
More info: While we are focused on remediation, users who have access, may be able to use Outlook on the web and Outlook mobile apps to connect the service.
Current status: We’ve identified an issue where users from Europe and United Kingdom are unable to connect to Exchange Online through the Outlook client. Our telemetry indicates that the service has now recovered but we are investigating the underlying cause and monitoring the service to ensure full recovery.
Scope of impact: Customers connecting to the Exchange Online service through our Europe and United Kingdom data centers may experience impact.
Accessing Exchange Online via other means such as Outlook for Web or a Mobile Device appear to be unaffected at this time.
15:00
We are aware of issues intermittently affecting users of Microsoft 365 to access Exchange Online (Email) via the Outlook Client.
Microsoft are already aware of the issue and are working on. Please see below update from Microsoft:
User Impact: Users may be unable to connect to the Exchange Online service using the Outlook client.
Current status: We’re investigating a potential issue.
Accessing Exchange Online via other means such as Outlook for Web or a Mobile Device appear to be unaffected at this time.
21:00
Microsoft Support now believe this issue is resolved. Please see the full update below:
User Impact: Users may have been unable to connect to the Exchange Online service using the Outlook client.
Final status: We’ve identified a Domain Name System (DNS) issue within components used to process the Outlook client connectivity to the service. This issue has self-recovered and after an extensive period of monitoring, we’re declaring the issue resolved.
Scope of impact: Impact was specific to users located in Europe.
Preliminary root cause: A DNS issue within front-end components, used to direct user requests, resulted in the Outlook client disconnecting from the service.
We are seeing service return to normal for our users. If you are still experiencing issues please get in touch through the usual support channels.
16:00
We have received a further update from Microsoft Support on this issue. Please see below:
User Impact: Users may be unable to connect to the Exchange Online service using the Outlook client.
Current status: We’ve identified a potential networking issue and we’re analyzing diagnostic data to determine the next troubleshooting steps.
Scope of impact: Impact is specific to users located in Europe
Accessing Exchange Online via other means such as Outlook for Web or a Mobile Device appear to be unaffected at this time.
15:00
We are aware of issues affecting users of Microsoft 365 to access Exchange Online (Email) via the Outlook Client.
Microsoft are already aware of the issue and are working on. Please see below update from Microsoft:
User Impact: Users may be unable to connect to the Exchange Online service using the Outlook client.
Current status: We’re reviewing system telemetry to isolate the source of the issue.
Scope of impact: Impact is limited to users located in Europe.
Accessing Exchange Online via other means such as Outlook for Web or a Mobile Device appear to be unaffected at this time.
12:00
TalkTalk have provided the following update:
“We are pleased to confirm that the earlier problem has now been resolved and access has been restored, once again we apologise for any inconvenience caused.”
If you are experiencing issues with your business broadband please get in touch – 01733830555
11:30
We have been made aware of issues with TalkTalk broadband across the UK.
At this time this seems to largely be confined to residential but we are monitoring with our suppliers.
Currently we do not believe any of our customers are affected.
If you are experiencing issues with your business broadband please get in touch – 01733830555
We are aware of issues with the O2 Network affecting some customers across the UK.
Please see O2’s Website for further information: https://status.o2.co.uk
17:30
Services should now have returned to normal.
The East coast Fibre restoration works have been completed and full resiliency across the supplier’s network has been restored.
All services should now be functioning as expected. Please contact the support team via the usual channels if you have any ongoing issues.
Thank you for your continued patience.
09:30
Services should now be starting to return to normal. Latest update from the supplier is below:
“The fibre issue in the London area has now been resolved as of 05:01, our monitoring and testing has confirmed that service is restoring across our platforms and we will continue to monitor network stability throughout this morning. We apologise for any disruption this incident has caused to your service. “
We have seen many customers services return to stability this morning and will continue to monitor this issue.
00:30
Issues are still ongoing. Latest update from the supplier is below:
“Work on the stabilisation of re-routed traffic across the temporary network connection remains a challenge and we have engaged our technology partner to work alongside us on this. This will continue to be a priority throughout the evening, but unfortunately we can’t yet provide an outline timescale on when this will complete.
Fibre engineers have identified the break between two fibre joints in the London area and are investigating the options to replace this with a local spare. Once we know the availability of a spare fibre we hope to be able to provide an estimated time to restore.”
The supplier’s engineers will be continuing to work on this throughout the night and will look to provide a full assessment and update at 08:00. They will still look to provide any material changes in status as they happen overnight but are looking to provide a full update across all areas at 08:00.
Until at least one of the Fibre breaks is restored service disruption is expected to continue.
Further updates are expected by 08:00
21:30
Issues are still ongoing. Latest update from the supplier is below:
“Fibre engineers are now on site and beginning work on the restoration. Our network team have been continuing the work on the network re-routing of voice and connectivity services and we expect to restore those services by 10pm tonight. We’re confident the re-routing work will fully restore services, however services remain at risk until the fibre restoration work is completed.“
Until at least one of the Fibre breaks is restored service disruption is expected to continue.
Further updates are expected by 23:00
20:45
Issues are still ongoing. Latest update from the supplier is below:
“We have isolated the fault to a dual fibre break on the East coast and the London area of our national network. Our technology partner has been engaged and they are deploying engineers to address both breaks. The incident continues to impact our voice and connectivity services and we are taking mitigating action at a network level to try to restore connectivity where we can.
We estimate engineers to arrive at the fibre break in the London area at 9pm at which point they can assess the extent of the work required to restore the fibre connectivity. Work will begin at 9pm on fibre break within the Leicester area, however we know the restoration work here is extensive due to an electrical incident in that area.”
Until at least one of the Fibre breaks is restored service disruption is expected to continue.
Further updates are expected by 22:00
17:15
The issues with Broadband and Phone lines for some customers are still ongoing.
The issues have been identified by the supplier to be down to a major fibre break in the supplier’s network infrastructure which is then causing large amounts of congestion across the network affecting Voice and Data connectivity.
Engineers are working to identify the cause of the break and repair it. In the meantime they are also working to route affected traffic over alternative routes to alleviate congestion and restore service.
Next update is expected by 18:30
16:30
The issues with Broadband and Phone lines for some customers are still ongoing.
The issues are still being investigated by the supplier’s engineers.
We are expecting a further update by 17:30
15:00
We are aware of issues across the country affecting Broadband and Phones for some customers.
Issues range from the following:
We are investigating these issues with the supplier and will update when we know more.
Further Update from Microsoft:
Current status: After further review of the captured data, we’ve determined that a recent authentication change has introduced a problem where users with multiple accounts are unable to authenticate to the service properly, preventing them from accessing their mailboxes. We’re working to develop a fix to address this issue and will provide an estimated timeline for mitigation early next week.
Scope of impact: Users may experience this issue if they operate multiple related and trusted Enterprise log-in accounts across Microsoft 365 and Windows.
Root cause: A recent authentication change has introduced a problem where users with multiple accounts are unable to authenticate to the service properly, preventing them from accessing their mailboxes.
Microsoft have also advised the following workaround may restore access to Outlook in the interim:
1) Close Outlook and other Office applications.
2) Open an Office desktop application, such as Word, PowerPoint or Excel.
3) Navigate to File > Account > Switch accounts, and then click the ellipsis next to the account that’s not having the problem.
4) Click Sign out and then OK. *Note: Users that have multiple accounts may find it difficult to determine which account to sign out of; though, they should be able to sign back in later if there is an issue with this process.
5) Close the Office application used for this process and re-start the Outlook desktop client.
6) When prompted for a password, click “sign in to a different account” and enter the correct information.
If the above does not work users will need to revert to using Webmail as previously advised.
We are aware of an issue affecting Users on Office 365 ability to use Outlook to send and receive email. Please see details from Microsoft below:
User Impact: Users with multiple enterprise sign-in accounts can’t connect to Exchange Online via the Outlook desktop client. Users experiencing this problem will see the Outlook desktop client showing as disconnected and may receive the following error when attempting to send mail: “Sending and receiving reported error (0x80040115). The connection to the Microsoft Exchange Server is unavailable. Outlook must be online or connected to complete this action.”
Users may be able to leverage one of the following options to circumvent impact for this event:
– Affected users may be able to use Outlook on the web or mobile clients to access their mailboxes.
If you are being affected by this issue please attempt to use Outlook Webmail for the time being until Microsoft resolve the issue.
To access Outlook for the Web browse to https://outlook.office365.com and login with your Office 365 Email Address/Username and Password.
06:20
2G, 3G, and 4G data back online.
If customers are still experiencing issues reconnecting to mobile data, please advise them to turn their handsets off and on again (or turn flight mode on and then off). This will refresh the service and trigger it to actively search for a data signal.
Please advise any customers experiencing slow or intermittent data access that this may be down to the large number of customers reconnecting to data at the same time.
Thank you for your patience and understanding.
08:37
The O2 network is experiencing an issue with DATA across the UK.
We apologise for any inconvenience and will share updates with you as soon as we have them.
Thank you for your patience and understanding.
10:15
We are aware of some data centre issues which may cause customers to have difficulty connecting. We are currently working on the issue.
15:12 pm
The supplier has now confirmed the issue has been resolved.
14:23 pm
We have been informed that the issue with slowness has now been resolved. We will continue to monitor the issue over time.
13:26pm
The supplier has identified the issue and is currently working on the resolution. We will continue to push updates once we have more information.
13:06pm
The supplier is currently investigating the slowness issues, we will update this when we have more information.
10:33am
As per our previous update power is restored and we believe all services are backup, if you experience further issues please contact our support desk.
08:59
We have been working on an issue in our data centre affecting cloud/hosted platforms. Power has been restored and we are working to get services back online as quickly as possible. Further updates within 2 hours.
12:52
The issue affecting some customers broadband services has now been resolved. If you still experience issues please contact the support team on 01733 667755.
10:02
The fault affecting customers has now been identified and is being worked on, we are hoping that services will be restored soon.
09:09
We are aware of a national broadband outage which is affecting some customers. The issue is being worked on, we are keeping up to date with developments and will update here within the next 2 hours.
09:25
We are aware of an issue on the iExchange platform which is causing a small number of customers to experience email issues, we are working on this issue as our top priority, we will update here..
|
10:18
The issue affecting connectivity for some customers is now resolved and all affected customer are now working normally.
09:00
We are aware of a connectivity issue affecting some of our customers, it is being investigate by our supplier and we will update as soon as possible. They have engineers working on it.
Symptons: Loss of internet and voice traffic
10:49
One of our third party hosted mail platforms is currently experiencing an outage which may result in some users not being able to send or receive email. This is being investigated as a priority, we will update progress as we receive it.
10:27
The Hosted Phone outage is now resolved and service should be restored to all customers. We are working on removing all diverts, this may take some time due to the volume.
08:52
We are aware of some issues with some customers hosted phone systems. Engineers are working on this in the data centre and we will provide an update here as soon as we have it.
If you require a divert please email support@greencityict.co.uk with the subject Hosted Divert, please include the number you wish to divert to.
14:34
We are aware of some issues with Microsoft Outlook which have been ongoing for some time. Microsoft has confirmed that some users are unable to send email or to access their accounts.
The issue appears to be intermittent and affecting some customers but no means all.
We will continue to monitor the situation and update this page as soon as we know more.
As this is an intermittent issue, if you are experiencing issues with Outlook please contact us by telephone on 01733 667755 so that we can investigate.
15:40
This issue has been investigated and resolved. There are a very small number of customers being affected, this customers are being support by our Engineers and Technicians.
08:58
We are aware of a broadband outage affecting some customers which we are currently investigating. Any updates will be posted here as soon as we have them.
The earlier issue was caused by a main network operation centre which suffered a power cut as of 9:13am due to external works in the local area. This also affected our ability to receive calls. Power is currently being provided via a back up service and recovery was from approximately 9:30am. There will have been some service interruption to voice and data services for a number of customers if their service was primarily hosted in this location, while services were rerouted to our resilient locations, and a number of services may still be recovering.
09:52
We are aware of a widespread telecoms issue and we are investigating. This issue is affecting us so please email you support request to support@greencityict.co.uk. Further updates will be available here as soon as we have them.
11:25
The MyDesktop Platform issue has now been resolved, if you are still experiencing issue please let us know in the usual way.
11:00
The issue affecting the platorm is still being worked on, further updates on this issue will be posted here
09:43
We are still working on the issue affected MyDesktop Platform customers, we will provide another update soon.
09:05
We are aware of an issue on the MyDesktop Platform this morning which is displaying as a certificate error.
The error appears to be preventing people being able to logon and connect to the system.
This is currently being worked on, we will provide a further update at 9:45am
Further updates can be found here.
14:43 – Resolved
The issue has been identified and resolved. We are currently contacting affected customers to ensure they are no longer experiencing issues.
The suppliers are aware of this issue and continue to work hard to resolve it.
11:43
We are currently aware of a call quality issue affecting some customers who have SpliceCom Hosted Phone Systems this is affecting external calls. We have notified SpliceCom Support who are investigating further with their carrier and are working to resolve the issue.
We apologise for any inconvenience caused. Please check back here for updates.
11:12
The issue regarding a national broadband outage appears to be resolved although the supplier fault has not yet been closed. Our customers are seeing services being restored, the restoration of services started approximately half an hour ago. Due to this being a national outage we may not receive an outage report, if we do it will be posted here.
09:39
We are aware of a major national broadband outage affecting businesses all over the country. We are working with our suppliers to resolve this issue as quickly as possible and will post updates here as often as possible.
If you require diverts on your phones please contact our help desk on 01733 667755.
13:18
The certificate issue on the mail platform has now been resolved.
The affected users Outlook clients should now be connecting and incoming and outgoing emails should be flowing (although this may take a little time). If you were an affected customer still experiencing issues with Outlook connecting close Outlook and re-open. If the issue persists please contact us on 01733 667755.
12:25
We are currently making progress on this issue and we believe mail to mobile devices has now been restored. We are continuing to work towards a full resolution for Outlook connectivity
11:02
We are still working on the issue affecting some customers using Outlook and hope to have this resolved very soon.
As previously mentioned Webmail is working for both sending and receiving emails using the following address https://mail.c1.questcloud.net/owa
Once again we apologise for the invconvenience
10:00
We are working on an issue with a corrupt certificate on one of our mail platforms which is causing Outlook not to connect for some of our customers. Webmail is still working by clicking through the certificate/security warnings.
We will post another update as soon as possible
Webmail address: https://mail.c1.questcloud.net/owa
We apologise for the inconvenience.
08:55
We are currently aware of a certificate error on part of our email platform that is preventing some users Outlook from being able to connect. Our engineers are currently working on resolving the issue at the moment. This issue only affects users connecting on Outlook, the Web mail portal is working without issue and users can send and receive email on that portal.
The Web mail address is https://mail.c1.questcloud.net/owa
Please check back here for updates
12:45
Apologies for the late notice but we have just received the notice below from our cloud service provider. There will be emergency maintenance late tonight to perform essential maintenance.
If you have any questions related to this please contact our helpdesk.
Effected systems: Virtual Private Cloud, questMail, VPS, questCloud Virtual Desktop
Description of works: We are currently experiencing an issue across our primary Hyper-V Clusters with Windows Remote Management Instrumentation (WinRM). This is effecting our ability to communicate with our Host Clusters and provide management access to customers. Some of the primary tasks WinRM provides are Virtual Machine load balancing across the clusters, Virtual Machine Migration, console management and storage management.
After consulting with the manufacturer of the software Microsoft we have been advised that the next course of action is to perform restarts of every node in our Host Clusters. Under normal operations this would not be an issue however, one of the effected systems of this fault is our ability to “Live Migrate” Customer Virtual Machines from one Host to another.
As a result of this completing this maintenance will effect service while each node is restarted.
Impact Details: Each Cluster Node will have its Virtual Machine guests placed into a “saved” or “paused” state then the node will be restarted and the guest Virtual machines resumed. During this time the Virtual machine will be unavailable, once the Virtual Machine is resumed it will continue from where it was before maintenance began without being restarted.
Service will be removed at 22:00 this evening and we are forecasting a restore of all services by 02:00, however it may be necessary to extend this until 04:00 in a worst case scenario.
We have been informed by City Fibre that between 06:00 and 15:00 on Sunday 27/11/16 Peterborough City Council are committed to completing the upgrade of their power distribution in the Town Hall. There should be no disruption of service, however due to the nature of the maintenance services are deemed “at risk” for the duration of this period. In the event of an outage and your service not automatically being restored you may need to restart both the white ONT & the supplied router or own equipment.
.
On Friday 4th November our office will be closing at 4pm.
Following the completion of our recent acquisition of Total Solutions and our continued dedication to providing the best service through a motivated and knowledgeable team we will be closing early on Friday 4th November 2016 for some team building.
It is important that all team members attend so from 4pm on Friday 4th November if you have an urgent support or maintenance issue please call in the usual way and follow the prompts, you will be asked to leave a message and one of our technical team will call you back shortly after the end of our meeting at 5pm.
If you have any other queries including non-urgent support queries from 4pm on Friday please email the relevant team or call on 01733 667755, leave a message and we will respond on Monday 7th November 2016.
If you have any questions please email getintouch@greencityict.co.uk
14.32 – The outgoing call issues have now been resolved.
We are aware of an issue affecting some customers when making outbound calls. When dialing out to some numbers some customers are receiving a number unobtainable tone. We are currently working with the carrier to resolve the issue as quickly as possible.