19:00
CityFibre have provided a further update on this issue and believe this issue now resolved following works yesterday and a period of monitoring:
Resolution Notes: Network Cards at two network locations replaced. The remaining two costed out links are internal to CityFibre and there is no impact to customers and will be forward managed separately
Resolved: 14:31 (Customer impact resolved post card Swap out at 19:25 on Saturday 23rd July).
We are also still seeing stability on affected circuits. As such we believe this issue now resolved.
If your service was affected and has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
16:00
CityFibre have provided a further update on this issue, please see below:
With errors continuing following the suspected faulty line card being replaced, focus has switched to another core device which connects directly to it. Joint investigations between CityFibre and the vendor of the other core device commenced a short while ago, and further alarms were quickly identified here.
The card causing the alarms was taken out of service in order for redundant links on another card to be forced into use. This will mitigate any remaining impact to services previously utilising that card and to prevent any further service disruption by our planned restoration activity.
The next stage is for engineers to attend the relevant location with replacement parts. The ETA for this activity is 18:30.
There is currently no customer service impact that we are aware of, with services either taking alternative routes round the network or utilising their designed service resiliency at this location.
We will provide a further update when we know more.
08:25
CityFibre have continued to work on this issue overnight, including sourcing replacement parts so they can swap out the affected card and arranging engineer resource to collect the parts and then attend site.
Currently ETA for Engineer to arrive at Slough DC is 08:30 this morning.
We will provide a further update when we know more.
21:00
CityFibre have provided a further update, please see below:
Engineering attendance has been requested to CityFibre Slough Datacentre to carry out a reseat of the card.
Once reseated they will remain on site for at least 30 mins to confirm success. If the issue manifests again then engineer will proceed with a Card Swap out.
Currently awaiting an ETA for engineers attendance to Slough.
We will provide a further update when we know more.
18:00
CityFibre have provided a further update, please see below:
Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.
It has been found that the origin point of the issue is on a line card situated within a core network device. During these investigations, two major actions have taken place in an attempt to restore service. The first action was to perform a soft clear on the line card itself which appeared to restore service for a short period before the issue resurfaced. And the second action was performing a full remote reboot of the affected line card.
During a period of monitoring, it has been seen that an issue has surfaced following the full reboot which is currently being investigated on a conference call between CityFibre NOC and our hardware supplier. Following on from the results of these investigations, the next course of action would be for engineers to be engaged to attend the site of the affected line card in order to perform a physical reseat of the line card itself.
Further to this, we are also considering a line card replacement should the issue resurface following a short monitoring period after the line card reseat.
We will provide a further update when we know more.
15:45
CityFibre have provided a further update, please see below:
CityFibre NOC have been investigating further and have determined that the issue is emanating from a specific location with the core network. As such, this device has been checked and the hardware supplier has been contacted directly for a Priority 1 case to be raised with them. During this time, CityFibre NOC have attempted to alleviate some issues across the core network by performing a workaround to reroute traffic to avoid the suspected affected device.
All of the requisite logs have been provided and CityFibre NOC are working alongside the hardware supplier to determine our next course of action. As of yet, it is still unclear as to when full service can be restored however we will endeavour to provide this as soon as we have it available.
We apologise for the disruption and hope to have a further update soon.
We have had reports that some services have stabilised since the changes were made by CityFibre to reroute affected traffic.
We will provide a further update when we know more.
14:00
We are aware of an issue affecting multiple CityFibre circuits nationwide that is causing partial or complete loss of service.
CityFibre are aware of the issue and have confirmed it is Major Service Outage and that they are investigating the cause.
Issues first started to manifest around 11:45 on a few circuits and have progressed since then to affect more circuits.
Below is the latest update from CityFibre:
Completed Actions:
Current Action Plan: At this juncture, it has been determined that several tunnels are currently flapping following initial investigations by CityFibre NOC.
We are working to determine as to why the tunnels are flapping in order for restoration works to begin. At this point, we are unable to provide an estimated time of repair however once one is available we will provide it as soon as possible.
We apologise for the disruption and hope to have a further update soon.
We will provide a further update when we know more.
19:30
After investigating CityFibre have identified the drop some customers experienced on 19/07/2022 appears to have been down to a brief power issue at an Openreach Exchange were some CityFibre equipment for the area is located.
No further issues have been seen since that appear to be related and so is believed to be an isolated incident at this time. As such we consider this issue resolved.
12:25
We are aware of an issue that caused a drop in service on some CityFibre circuits in Peterborough between around 11:30 and 12:15 today.
We have raised this issue with CityFibre to advise what happened and will provide an update when we know more.
Services on the whole now appear to have recovered, but if you are still seeing a loss of service please reboot any relevant equipment in the first instance and if still not restored please get in touch through the usual channels.
17:00
Good Evening
Below is the latest update on this issue received 27/06/2022 PM from CityFibre. We have not shared this update before now due to some incidents that seemed possibly related that we have been waiting to receive clarity on before sharing this update and closing down the major incident ticket on our end.
We are now confidant the other incidents we observed are NOT related to this MSO with CityFibre but are in fact isolated incidents caused by separate issues/circumstances. Outside of these isolated incidents we have seen no evidence of reoccurrence this week.
As such we believe this issue now resolved.
Resolution Notes:
Following a 5 day period of extensive monitoring, no further impact has been experienced since approx. 12:00BST on the 22nd June 2022 when the MPLS path between Reading – Coventry was costed out of the network. CityFibre continued enhanced monitoring and engagement as we build confidence that our stability is restored.
Investigations into the root cause continue via City Fibre’s Problem Management process. CityFibre NOC and Network Engineer teams continue to liaise with Vendors Nokia and Ciena regarding root cause investigations.
Resolved: 22/06/22 12:00
If your service was affected and has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
18:45
Please see below the latest update provided by CityFibre on this issue:
CityFibre NOC Engineers have continued to monitor the network and no further impacts have been experienced. Service has remained stable since approx. 12:00BST on the 22nd June 2022.
The relevant captures remain in place on recommendation from the cross-party vendors to assist us with our investigations and in the event of a further failure use the logs to determine any potential software bugs across the LSP’s.
Nokia believe there may be a software big on their element of the network, however, the continue to investigate and will provide a status update week commencing the 27th June 2022.
The MPLS path between Reading – Coventry remains costed out of the network and this will be the condition until we have a plan of attack to mitigate further impact to service.
The incident remains on extensive monitoring and the Major Incident Team remain engaged to assist in the event of a reoccurrence.
We will update you further when we know more.
22:00
Please see below the latest update provided by CityFibre on this issue:
CityFibre NOC Engineers have continued to monitor the network and no further impacts have been experienced. We have engaged with a number of our customer who have confirmed service stability.
At this juncture we have been unable to determine to underlying root cause of the incident, however, we have all the relevant captures in place on recommendation from the cross-party vendors to assist us with our investigations and in the event of a further failure use the logs to determine any potential software bugs across the LSP’s. CityFibre, in conjunction with our vendors, continue to investigate this issue as priority, however, there is potentially several contributing factors.
The MPLS path between Reading – Coventry was removed from the network as part of our investigations and this has helped stabilise services since around 12:00 BST on 22nd June 2022. The network will remain in this situation until we are confident that we have an action plan to mitigate further impacted to service.
Our Technical Teams continue to pursue the live TAC cases with the vendors to (i) repair this path and (ii) determine root cause and remediate the behaviour observed on certain MPLS paths. A further Major Incident Meeting has been scheduled for the 24th of June and the Major Incident Team will release a further statement at 17:30 on the 24th of June 2022.
The incident remains on extensive monitoring and the Major Incident Team remain engaged to assist in the event of a reoccurrence.
Next Update: 17:30 24/06/2022
We will update you further when we know more.
12:30
Please see below the latest update provided by CityFibre on this issue:
CityFibre, in conjunction with our vendors, have continued to investigate this issue with priority. The current assessment is that there are several contributing factors. CityFibre have removed an MPLS path between Reading – Coventry during the course of investigations and this has helped stabilise services since around 12:00 BST on 22nd June 2022. We are still monitoring the effectiveness on this action.
Our Technical Teams continue to pursue the live TAC cases with the vendors to (i) repair this path and (ii) determine root cause and remediate the behaviour observed on certain MPLS paths. We expect to release a further statement at 17:30
Next Update: 17:30
We will update you further when we know more.
23:30
Please see below the latest update provided by CityFibre on this issue:
CityFibre in conjunction with the vendor have completed their Technical conference call and have redefined the debug setup in the event of further LSP flaps. Following the meeting and recommendation from the vendor CityFibre have implemented additional captures across elements of our core network to capture more detail in the event of a reoccurrence. The technical session covered the detail below, however, at this juncture the underlying root cause has not been identified therefore we have no recovery plan.
• Reviewed the historical debug captures – no issues identified
• On vendor recommendation CityFibre have applied further debug captures on more nodes
• CityFibre network has been more stable today
• Amended the way the debugs are being stored to remove limitations
• Behaviours seen on another element of the network and an incident has been raised with Juniper. Feedback will flow into Nokia when we have the feedback.
At this juncture we have exhausted all avenues of investigation and CityFibre NOC continues to monitor the network for further LSP flaps. Nokia remain on point to support us from a technical perspective and review the analysis as we experience issues across the network.
CityFibre NOC continues to monitor the situation and the MI Team will reconvene tomorrow to review the status of the network overnight.
Next Update: 11:00 23/06/2022
We will update you further when we know more.
14:00
Please see below the latest update provided by CityFibre on this issue:
A conference call has taken place between CityFibre and Nokia at senior level at we have agreed an approach to assist with investigations as outlined below:
• Technical Call to be convened with both teams to look at end to end (ETE) path for LSP Flaps
• CityFibre to investigate a Nokia ETE path and pass to the NOC to monitor for LSP Flaps to rule out multi-vendor associations
• Daily management bridge with status updates arranged for 17:00
• Review the current software and devise a plan to upgrade to the latest version
• CityFibre NOC to send any further drops to Nokia
CityFibre are confident that we now have the right level of support and focus from a Technical & Escalation perspective and updates will follow following todays call.
Next Update: 18:00
We will update you further when we know more.
08:30
Please see below the latest update provided by CityFibre on this issue:
CityFibre have continued to escalate the incident with the vendor, however at this juncture we have no further information in terms of the recovery plan from the vendor. Escalation into the vendor is now at Vice President level and we have asked them for a Technical Session to devise a plan of action to alleviate impacts across the CityFibre network on the 22nd June 2022.
We apologise for the inconvenience caused. Our main priority is to deliver the levels of service that our customers deserve. CityFibre are working to affect a full restoration as soon as possible.
Next Update: 10:00 22/06/2022
We will update you further when we know more.
16:25
Please see below the latest update provided by CityFibre on this issue:
CityFibre have held a further conference call with Nokia who have confirmed they have received the outputs from the outages this morning and are analysing the logs and anticipate to provide feedback by 22:00.
Our Major Incident Team are escalating into Nokia for further support at a senior level and escalations will continue.
We will provide a further update from the outputs from Nokia.
Next Update: 22:30
We will update you further when we know more.
16:30
Please see below the latest update provided by CityFibre on this issue:
A number of CityFibre customers have been experiencing intermittent loss of service from the 10th of June 2022. CityFibre in conjunction with our vendor (Nokia) have been gathering evidence of the drops from the impacted customers to assist with investigations. At this juncture the vendor has been unable to determine the cause of the incident and investigations continue, however, they have confirmed the incident conditions are not linked to any hardware.
CityFibre NOC Engineers continue to run a technical support file/debug every hour to assist with investigations on recommendation from the vendor to provide further evidence of the drops in order for them to take remediation to mitigate impact.
Identifying the cause of the incident has been problematic and CityFibre are working with the vendor to establish the correct evidence of the drops. Escalations have been placed into the vendor at a Senior Level and a further call has been scheduled to review the status and output from further drops at 14:30 on the 21st of June 2022.
To allow us to gather the evidence to pass to the vendor to assist with investigations the MI Team and vendor will reconvene on the 21st June at 14:30 and an update will follow.
• CityFibre NOC to monitor and feedback to vendor • CityFibre Technical Support Centre to continue to triage incidents on report • CityFibre TSC to link any associated incidents • CityFibre Major Incident (MI) Team – Nokia to reconvene on the 21st of June
Next Update: 15:30 21/06/2022
We will update you further when we know more.
10:15
Please see below the latest update received from our supplier on this issue:
Following the call between our NOC team and our hardware supplier Nokia, the NOC team have performed debug operations and have been able to capture some of the requisite information required for Nokia to analyse.
This is being sent over to Nokia currently and we are ensuring that our NOC staff in overnight and at the weekend will be reviewing and assisting the investigations where possible.
Currently, we do not have a timeframe on when this information will be fully analysed however we will endeavour to provide updates as and when they become available. Should we not have any direct updates however, please be advised that we will be scheduling our next update for Monday 20/06/2022 at 10:00
Next Update:
10:00 20/06/2022 pending any developments with our Hardware Supplier Nokia.
Given the times mentioned above, we are expecting a further update later this morning but there is usually some delay for updates to filter down to us.
We will update you further when we know more.
12:00
Overnight CityFibre Changed the optics on the link between London and Cambridge – However we can see the network drops are still occurring.
The latest update from CityFibre is below:
We understand your frustration on this intermittent issue and please accept our apologies. The latest update is the following:
CityFibre has multiple teams working on this issue from the Network Operations Centre (NOC) and Architecture & Engineering.
CityFibre has a critical case open with Nokia.
One of the nodes in question is a core node in the CityFibre network within the London area and we cannot isolate or cost out traffic as the intermittent issues seems to be spread over multiple tunnels.
Rest assured this issue has the visibility within CityFibre and our vendor to drive resolution
We will update you further when we know more.
18:00
We have received an update from the supplier that a change is scheduled for 22:00 this evening on what is believe to be the affected link to address this issue.
As such we will be monitoring post this change for any further dropouts on our monitoring so we can make the supplier aware if we do see any.
We will update you further when we know more.
14:45
We have received a further update from the supplier on this issue:
CityFibre NOC continue to liaise with Nokia. Nokia continue focusing on node – lon5-1. The Debug output was provided to Nokia and we are currently awaiting their feedback.
Ciena is still engaging Tier 4 to investigate this issue.
We will update you further when we know more.
09:30
Works to resolve this issue remain ongoing and below is the most recent update from CityFibre following yesterday’s investigations:
Our night shift NOC team has informed us that it looks like the LSP flapping occurred again overnight at just after 02:00. It was also found that the LSP appears to flap once the physical flaps, the LSP tunnels follow suite.
An update from Ciena has been chased throughout the night, however we have received requests from Nokia (hardware supplier for core devices) to run some debug commands as well as answer a few of their questions with the assistances of our Architecture and Engineering team.
I have asked for this to be liaised with them as a matter of urgency given the matter at hand. This will be the main focus for today as to ensure that we have as much details to speak to Nokia with as possible.
We will update you further when we know more.
08:45
We are aware of an issue causing intermittent short drops in service for a small number of CityFibre services usually lasting 1-2 minutes. We first noticed this issue over the weekend and have been investigating with CityFibre since who are working on a resolution.
In most cases where we are seeing a drop if a backup broadband circuit is in place it is enough to trigger a failover event.
We will update you further when we know more.
10:20
We have received a further update from the supplier who believes this issue now resolved:
The switch is now back from the reboot and we are seeing traffic and routing back as it should be – we will investigate further the cause here as a background task to ensure it does not re-occur.
We did see some services drop due to the reboot and then come back up.
If your service was affected and has not been restored as of yet please reboot any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels
09:45
We are aware of a possible issue that may affect some services with one of our suppliers:
We are currently seeing an issue with some traffic routing via Telehouse West – we currently believe one of our core switches has a forwarding issue,
Our NOC team are currently working to resolve this which MAY involve a brief reboot of the affected switch – causing a short downtime for services directly connected to that switch
We apologise for the brief and short notice reboot but want to resolve the issue as quickly as we can
We will update you further when we know more.
20:45
We have been monitoring services over the weekend and today and so far the network has remained stable.
The supplier remains cautious and will be continuing to monitor the network for any further issues for at least the next few days.
We are aware of a small number of websites that for certain customers are not accessible and are working with those customers and the supplier’s NOC team directly to resolve.
If your service was affected and has not been restored as of yet please reboot any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels
16:45
We have received the below update from the supplier:
A considerable amount of changes and work has been completed during the day today including additional upstream capacity and changes to the filtering to protect our network. There is ongoing changes and updates ongoing this evening to bring the network stability back to what we would expect to provide our customers with.
Our technical team have also been in touch with the supplier directly and understanding more about the work still ongoing believe issues should be largely resolved this evening assuming all goes well.
We will be monitoring this issue this evening and will provide further updates as they become available.
Once again we apologise for the ongoing issues.
13:30
To the follow-up from the drop this morning, this is currently being worked on by the supplier’s senior team, the up-stream NOC team & the transit providers to ascertain what occurred and resolve fully.
In the meantime to get things back to a working state for most customers they have had to roll back some of the changes they’d previously made which has effected some of the mitigations for SIP and VPN traffic. They are working on restoring these as quickly as possible.
We are aware some customers still have lingering issues outside of SIP/VPN traffic and these are being looked into on a case by case basis.
We will update further when we know more and apologise for the ongoing issues.
09:45
We are aware another drop on services has just occurred.
We have reported this to the supplier and are waiting for an update as to what has caused it. We will update further when we know more.
Once again, we apologise for these ongoing issues.
08:00
The last update we received on this from the supplier last night indicated work was still ongoing to re-route affected traffic with the upstream transit provider overnight and to expect a further update this morning.
As soon as we have received this we will update further. Please note there may be some delay between us receiving an update from a supplier and being able to pass it on to customers.
18:45
Please see most recent update from supplier below on this issue:
Good evening. We are are working with the senior teams at the up-stream provider who have moved a large amount of the traffic that was being affected away from the routes. We are seeing improvements in the situation but we are also aware that some of our customers are still impacted.
We will update further when we know more. Please note there may be some delay between us receiving an update from a supplier and being able to pass it on to customers.
16:30
We are aware of some services still experiencing lingering issues and the supplier has confirmed they are related to this outage. They are working on trying to resolve these remaining issues and have provided the following update:
We have seen some improvement to services since our last update but just to confirm that we are still working on mitigation of the transit problems with the upstream provider; once we have a better view on timings etc we will add a further update here and in any event, one at 6PM.
We will update further when we know more. Please note there may be some delay between us receiving an update from a supplier and being able to pass it on to customers.
15:15
We are aware of some services still experiencing lingering issues and the supplier has confirmed they are related to this outage. They are working on trying to resolve these remaining issues as soon as possible.
We will update further when we know more.
14:00
Within about 45 minutes of issues reoccurring we saw them settle back down again and services begin to return to normal, we are currently monitoring whilst we await a further update from the supplier.
We will update further when we know more.
12:30
Unfortunately we have seen a re-occurrence of this issue on our monitoring and the supplier has confirmed it appears the same upstream transit provider is once again having issues:
Our apologies; one of our up-stream transit providers is warning of route instability again. We were told this had been resolved but are actively working with them to understand why this had recurred and mitigate the cause. Our sincere apologies.
We will update further when we know more.
20:30
We have received a further update on this issue from our suppliers who have confirmed this should now be resolved. They are seeing traffic flowing normally again and are not seeing any lingering issues. From our own limited testing we also are not seeing any lingering issues.
As mentioned previously this issue was caused by a fault with an upstream transit supplier which affected multiple services.
If your service was affected and has not been restored as of yet please reboot any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
16:30
We have received a further update on this issue and work is still ongoing to fully resolve and re-route affected traffic. Most services have come back online with minimal lingering issues and those that are still affected are largely seeing only intermittent packet loss. Currently whilst a firm ETA has not been given, we have been told to expect issues to be fully resolved sometime this evening and will be continuing to monitor the situation.
Issues may manifest as intermittent or permanent loss of service. In the case of Phone services this could also manifest as poor call quality.
We will update you further when we know more.
15:00
We have received a further update on this issue and now know the issue is with an upstream transit provider affecting multiple services. Suppliers are working to mitigate and route around. This is why the issue appears to be intermittent and for some customers is no longer affecting them.
Issues may manifest as intermittent or permanent loss of service. In the case of Phone services this could also manifest as poor call quality.
We will update you further when we know more.
14:30
We are aware of an outage affecting various internet based services since about 14:00.
The supplier is aware of the issue and engineers are engaged to work on a resolution.
Below is a list of some of the services that may be affected:
Remote Desktop/MyDesktop Services.
Hosted Phones.
Broadband.
Web Hosting.
Issues may manifest as intermittent or permanent loss of service. In the case of Phone services this could also manifest as poor call quality.
We will update you further when we know more.
Please be aware of upcoming maintenance that affects your service:
Service: CityFibre
Customer Impact: Service Affecting
Location: Peterborough
Reason for Maintenance: To resolve ongoing network issues supplier has been experiencing in Peterborough following guidance from the hardware vendor.
Description of Works: To upgrade and replace Core Router PB1244 in Peterborough.
Start Date and Time of Maintenance: 09/06/21 01:00
End Date and Time of Maintenance: 09/06/21 06:00
Time of Maintenance: 5 Hours.
Additional Info: Customers will experience an up to 5 hour outage during the maintenance window.
If your service is affected and is not restored following the completion of the maintenance please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
Please be aware of upcoming maintenance that affects your service:
Service: CityFibre
Customer Impact: Service Affecting
Location: Peterborough
Reason for Maintenance: This is required due to an integrated port not being operational.
Description of Works: To replace Core Router PB1244 in Peterborough.
Start Date and Time of Maintenance: 10/05/21 01:00
End Date and Time of Maintenance: 10/05/21: 05:00
Time of Maintenance: 4 Hours.
Additional Info: Customers will experience an up to 4-hour outage during the maintenance window.
If your service is affected and is not restored following the completion of the maintenance please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
23:30
We have received a further update from the supplier who now believes this issue fully resolved, please see below:
Once again, please accept our apologies for the problems you’ve seen over the previous couple of days. We realise that this has caused you all serious issues and for that, we’re very sorry.
Various internal changes have been implemented over the last 48 hours and currently, we believe that the network and associated services are now stable and will remain that way. We continue to monitor the situation closely to ensure that our network remains stable and there’s no further impact to your services.
Currently from what we can see services appear to have returned to normal and look to be stable and have been for some time now. This issue as previously mentioned was due to a large scale DDoS (Distributed Denial of Service) attack against one of our supplier’s networks.
If your service was affected and has not been restored as of yet please reboot any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
16:30
We are once again seeing services start to settle down again and stabilise.
We are continuing to monitor the situation.
We will update here further when we know more.
15:30
We are aware that after a period of relative stability over the last few hours another drop has occurred.
The supplier is aware and working to resolve.
Unfortunately the DDoS attack that is the cause of these issues has not stopped in its intensity since it began yesterday and as a result the increased load on the network is causing intermittent drops and issues.
We will update you as soon as we know more.
10:00
We are aware service has gone down again and the supplier is investigating:
We are aware of issues still ongoing; we are actively working on these as before to mitigate their effects. We don’t have an ETA at the moment but we will update as soon was we do.
We will provide further updates as soon as we know more.
09:00
The supplier continued to apply mitigations works overnight and are monitoring the situation today. So far this morning we are seeing services remain stable.
A further update is expected by midday.
We will update further when we have received this.
16:45
We have received a further update from the supplier on this issue.
The DDoS attack itself is still occurring but the supplier is continuing to implement remediation works to reroute traffic and alleviate the load on the network. As such we have seen a number of services come back online however issues are still expected until the DDoS is fully mitigated.
We will continue to provide updates as we know more.
15:00
We are aware this issue remains ongoing and the supplier is working on resolving.
We apologise for the delayed update, the outage is affecting our access to certain services and systems and we are working on putting workarounds in place where possible to keep everyone up to date.
Our support desk ticketing system remains largely unaffected at this time but there is a backlog of requests from the outage slowing our responses there and we are doing our best to clear this backlog and update everyone.
We will update when we know more.
14:00
The supplier has advised they now believe this issue resolved and we are seeing services return to normal for affected customers.
Please see below update from the supplier:
We are seeing most services recovering. The attack target remains offline but we believe that this incident is now contained. We apologise again for this interruption in service. If you are still seeing issues, please restart your equipment.
This issue as previously mentioned was due to a large scale DDoS (Distributed Denial of Service) attack against one of our core supplier’s network.
If your service was affected and the issue is not resolved please get in touch with our support team via the usual channels.
13:35
We are aware of an outage affecting various internet based services. This is due to a large scale DDoS (Distributed Denial of Service) attack against one of our core supplier’s network.
The supplier is aware of the issue and has provided the following update:
We are mitigating a large portion of this attack traffic but currently, the transit links remain saturated which is causing the current ongoing problems. We continue to work to resolve this as quickly as possible & apologise for the ongoing inconvenience caused.
We will update you further when we know more.
13:00
We are aware of an outage affecting various internet based services. This is due to a large scale DDoS (Distributed Denial of Service) attack against one of our core supplier’s network.
The supplier is aware of the issue and engineers are engaged to work on a resolution.
Below is a list of some of the services that may be affected:
Remote Desktop/MyDesktop Services.
Hosted Phones.
Broadband.
Web Hosting.
Issues may manifest as intermittent or permanent loss of service. In the case of Phone services this could also manifest as poor call quality.
We will update you further when we know more.
10:30
We have identified the cause of the issue and put a fix in place. We are now seeing users able to connect without issue.
As such we believe this issue to now be resolved. If you are still experiencing issues please contact support via the usual channels.
09:30
We are aware of an issue preventing some MyDesktop customers from logging on this morning.
Engineers are investigating the cause of this issue and we will update here when we know more.
CityFibre have advised this maintenance did not go ahead as planned due to unforeseen circumstances and has now been rescheduled as per below:
Service: CityFibre
Customer Impact: Service Affecting
Location: Peterborough
Reason for Maintenance: Due to ongoing issue in Peterborough, maintenance is required to bring stability to the network.
Description of Works: Reboot of Peterborough PB1244 Core Router.
Start Date and Time of Maintenance: 26/03/21 00:01
End Date and Time of Maintenance: 26/03/21: 03:00
Time of Maintenance: 3 Hours.
Additional Info: Customers will experience approx. 30 minute outage during the maintenance window.
If your service is affected and is not restored following the completion of the maintenance please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
Please be aware of upcoming maintenance that affects your service:
Service: CityFibre
Customer Impact: Service Affecting
Location: Peterborough
Reason for Maintenance: Due to ongoing issue in Peterborough, maintenance is required to bring stability to the network.
Description of Works: Reboot of Peterborough PB1244 Core Router.
Start Date and Time of Maintenance: 25/03/21 00:01
End Date and Time of Maintenance: 25/03/21: 03:00
Time of Maintenance: 3 Hours.
Additional Info: Customers will experience approx. 30 minute outage during the maintenance window.
If your service is affected and is not restored following the completion of the maintenance please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
23:00
CityFibre have advised the fault has been fixed and this issue should now be fully resolved:
We have had confirmation that this issue is now resolved and that services are no longer at risk. Following assessments by our engineering resource at multiple locations, the damage previously mentioned was unrelated and not the cause of the issue on the remaining impacted service. Instead, a fibre microbend was identified and corrected, with testing demonstrating expected results.
If your service was affected and has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
17:15
We are aware of an issue affecting some CityFibre circuits across the UK causing either intermittent service drops or total loss of service.
CityFibre are aware of this issue and have provided the following update:
After further optimisation of the network we are seeing restoration to the affected customers.
As the there is still an open incident on the Leeds to Sheffield link, services should be considered at risk until the link has been brought back into service.
If your service was affected and has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance.
16:15
We are aware of an issue affecting some CityFibre circuits across the UK causing either intermittent service drops or total loss of service.
CityFibre are aware of this issue and have provided the following update:
After the initial re-routing we are still seeing a portion of customers impacted by the major incident. Further work is being carried out in order to optimise network traffic and alleviate the issues being experienced. We also continue to press our third party supplier for updates on the failure on the link between Leeds and Sheffield. Further update will follow at 17:00
We will update here as we know more.
15:45
We are aware of an issue affecting some CityFibre circuits across the UK causing either intermittent service drops or total loss of service.
CityFibre are aware of this issue and have provided the following update:
The incident is related to an issue on a link between Leeds and Sheffield. Third party providers have already been engaged and are investigating. In the meantime, to mitigate the impact we have re-routed traffic over an alternative link. We are now in the process of assessing the impact of the re-routing. A further update will follow at 16:00.
We will update here as we know more.
15:15
We are aware of an issue affecting some CityFibre circuits across the UK causing either intermittent service drops or total loss of service.
CityFibre are aware of this issue and have engineers investigating. Initial diagnostics have confirmed this is not specific to any one geographic location.
We will update here as we know more.
14:30
We are aware of a brief issue that affected some services hosted in our Datacentre.
This appears to have been a routing issue with one of the links into the Datacentre and is already resolved.
We are monitoring for any further issues.
14:30
We have discussed this issue further with CityFibre and received clarity on the workaround that has been put in place.
After extensive testing they believe the issue appears to be with some of the Fibre ports on one of the Core Routers itself and so they have moved the effected links to alternative known good ports on the router. As such there should be no noticeable impact to service or reliability going forward from this.
CityFibre will continue to investigate the problematic ports and will look to reintroduce these into production at a later date if/when the issues with them have been resolved and extensively tested prior to doing so. This will be part of planned scheduled maintenance at a later date.
As such this issue is considered resolved.
07:45
CityFibre have managed to restore service and have provided the following update:
Engineers have restored customer services although this is only a temporary solution at the moment as a workaround is in place. Further investigation work will be undertaken and a permanent solution will be introduced through our change management process. We will update when this will happen. Apologies for the issue and the disruption this will have caused.
We have requested from CityFibre more information on the temporary solution put in place and if customers should expect further issues whilst the permanent solution is being worked on or not and we will update here if we hear back.
We can see many of the services that we monitor are now back online and have been since the early hours of this morning. Failover circuits where present have failed back across successfully.
If your service was affected and has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
00:15
CityFibre have confirmed there is a small number of circuits with issues in Peterborough today resulting in a total loss of service. They believe this to be a Fibre issue and Engineers are on site:
Engineer parts have arrived on site and further work is ongoing, alongside our NOC team.
Further updates will be provided, once we have any new information from the engineers.
We will update here when we know more
21:45
CityFibre have confirmed there is a small number of circuits with issues in Peterborough today resulting in a total loss of service. They believe this to be a Fibre issue and Engineers are on site:
Replacement optics have been sourced that are compatible with both Cisco and Nokia (MSA compatible). The link from PB1244 – PB1344 is 12km, these optics are 80km so have requested attenuators to align the link up with the required light level.
Engineer resources have been arranged to collect parts and attend both sites.
Further updates will follow once parts and engineer resources have arrived on-site.
We will update here when we know more.
20:10
CityFibre have confirmed there is a small number of circuits with issues in Peterborough today resulting in a total loss of service. They believe this to be a Fibre issue and Engineers are on site:
Engineer resources have confirmed no lights within the fibre is being received between PB1344 and PB1244. They have also proved that the port is up which suggests the SFP is not related to the issue and lies within the fibre.
Further updates will follow once the engineer resources have reached PB1244.
We will update here when we know more.
19:10
CityFibre have confirmed there is a small number of circuits with issues in Peterborough today resulting in a total loss of service. This appears to be the same customers that were affected by a previous fault. They believe this to be a Fibre issue and Fibre Engineers are on site:
Engineer resources are on-site who continue to investigate the issue. We currently have fibre engineers and hardware teams present working alongside our NOC team.
We will provide a further updates in the next hour.
We will update here when we know more.
17:50
CityFibre have confirmed there is a small number of circuits with issues in Peterborough today resulting in a total loss of service. This appears to be the same customers that were affected by a previous fault. They believe this to be a Fibre issue and Fibre Engineers are now on site:
Engineer resources have now arrived on-site where investigations have commenced. We will provide further updates in the next hour.
We will update here when we know more.
16:45
CityFibre have confirmed there is a small number of circuits with issues in Peterborough today resulting in a total loss of service. This appears to be the same customers that were affected by a previous fault. They believe this to be a Fibre issue and are arranging Fibre engineers as per below:
Our NOC team have continued to investigate and believe the issue is fibre related rather than hardware. They have spoken with our ERS contractor in advance of them attending site and to ensure that they are aware of the actions needed. We have also engaged with our first line maintenance teams in case anything additional is needed and have them on standby. As soon as we are able to confirm planned arrival on site, we will update.
We will update here when we know more.
14:45
CityFibre have confirmed there is a small number of circuits with issues in Peterborough today resulting in a total loss of service. This appears to be the same customers that were affected by a previous fault. Engineers are on-route to Peterborough as per below:
Our Team have advised that it is the same link at fault as earlier in the week. However, the fault conditions are different, so it does not appear to be a repeat fault. We have undertaken several steps to clear any issues with our hardware and cards, and we can see that other ports etc. are up and running. We’ve interrogated the logs and events and can see nothing untoward other than a loss of light from one side. Therefore, we are arranging engineers to attend the site and will update with and ETA as soon as we are able.
We will update here when we know more.
14:05
We have had alerts on our monitoring of several CityFibre circuits going offline in Peterborough in the last 15 minutes.
This issue has been raised with CityFibre to investigate further.
We will update here when we know more.
07:35
CityFibre have advised work to replace the faulty card was completed overnight and that this issue should now be resolved:
03:25: Engineer is on site and about to commence work. Update will follow upon completion of the task.
04:55: The card has been replaced, and we have collected further diagnostic information. We will confirm operational status once this has been reviewed by our vendor.
05:25: We believe this issue to now be resolved, and request that any affected users retest their service. Please accept our apologies for any inconvenience that may have been caused.
We can see many of the services that we monitor are now back online and have been since the card swap took place. Failover circuits where present have failed back across successfully.
If your service was affected and has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
00:15
CityFibre have advised that the maintenance has completed successfully however it has highlighted another potential faulty component that is still causing issues for some circuits and they will continue to work on this overnight:
Our vendors are dispatching an engineer to replace a suspected faulty card in our PB1344-rtr-1 node. ETA is 02:30.
All circuits that were online prior to the maintenance we believe should now be back online. If this is not the case please reboot your CityFibre ONT and any related CPE in the first instance. Circuits that were down prior to the maintenance will still be affected by the above issue that is being worked on.
We will provide a further update here in the morning.
16:45
CityFibre have issued a notice that they will be performing emergency maintenance this evening related to this issue:
Tuesday 23rd February 2021, 19:00 – Tuesday 23rd February 2021, 23:59
During the above window, we will be completing emergency maintenance to upgrade and replace our core router at the above location. This is required in order to resolve the issues we have been experiencing in Peterborough. There will be a small window of downtime as we replace the chassis and migrate services across.
This will affect ALL CityFibre services in Peterborough.
16:05
CityFibre have confirmed there are a small number of circuits with ongoing issues today and that they are still investigating:
Engineer resources have now arrived on-site where investigations have commenced. We will provide further updates in the next hour.
We will update here when we know more.
14:15
CityFibre have confirmed there are a small number of circuits with ongoing issues today and that they are still investigating:
Based on NOC troubleshooting we have engaged engineer resources to attend the affected site and carry out further testing.
Engineer resources have been arranged with an ETA of 16:00. We will provide a further update upon the engineers arrival.
We will update here when we know more.
12:25
CityFibre have confirmed there are a small number of circuits with ongoing issues today and that they are still investigating:
We are still investigating this issue with our senior agents within the NOC team and will provide further updates in the next hour.
We will update here when we know more.
10:45
CityFibre have confirmed there are a small number of circuits still with lingering issues this morning and that they are investigating:
Following the outage last night affecting Peterborough, we are still aware of a small number of circuits still down. Our NOC team are currently investigating the situation and we will provide a further update in the next hour.
We will update here when we know more.
23:55
CityFibre have confirmed all circuits should be back online and this issue should now be resolved.
We have requested a full RFO (Reason for Outage) Report for this issue and will make this available to our customers upon request when we have received this.
If your service was affected and has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
23:15
CityFibre have advised they believe all circuits should be back online however we are still seeing several circuits that are down and are querying this with them.
We will update here when we know more.
22:45
We have seen on our monitoring a few circuits come back online but not all.
No official update from CityFibre as of yet.
We will update here when we know more.
22:15
CityFibre have advised the additional engineering resource has arrived and is working on the issue with the engineers already on site.
Further updates are expected after 23:00.
We will update here when we know more.
21:45
CityFibre have advised after initial diagnostics by engineers on site further engineering resource was required.
This engineering resource was expected to arrive by 21:35.
We do not as of yet have any further update.
We will update here when we know more.
17.15
We have had the following update from CityFibre:
Engineer resources have been engaged to attend the affected site with an ETA of 19:00. Further updates will be posted upon the engineer’s arrival.
We will update here when we know more.
16:25
We have had alerts on our monitoring of several CityFibre circuits going offline in Peterborough in the last 30 minutes.
This issue has been raised with CityFibre who advised they are aware of the issue and are already investigating.
We will update here when we know more.
18:00
CityFibre have confirmed all circuits should be back online and this issue should now be resolved.
We have requested a full RFO (Reason for Outage) Report for this issue and will make this available to our customers upon request when we have received this.
If your service was affected and has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
17:15
All CityFibre circuits that went down between 16:30 – 17:00 appear to now be back online.
We have chased CityFibre for an explanation and further update on the main fault and will update here when we know more.
As mentioned previously the brief outage just now was not planned so we could not warn anyone ahead of time. We will be querying why this was the case.
If your service was affected and has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
16:30
We are aware some CityFibre circuits in Peterborough that were up have just gone down again.
A CityFibre engineer was due onsite in Peterborough at 16:00 and we believe this is most likely related but we are attempting to confirm with CityFibre as they have not advised us of any expected downtime for working sites.
We will update here when we know more.
14:30
CityFibre have advised they are still seeing issues in certain areas of the UK.
They have provided the following update:
We are seeing the majority of services now restored apart from those directly connected through Peterborough. We are now awaiting an engineer to attend Peterborough sites with an ETA of 16:00, the technical teams will then work with the engineers to restore the services connected to Peterborough sites.
We will update here further as soon as we know more.
13:30
CityFibre have advised they are still seeing issues in certain areas of the UK.
They have provided the following update:
Work has been undertaken to return stability which involved shutting down one of our links. Following this, we have seen some services restored however, customers in the Peterborough area may still be impacted. We are currently waiting for engineer resources to attend Peterborough and carry out further troubleshooting.
Once Peterborough services have been restored we will then look to put a permanent fix in place ensuring any changes goes through the change management process due to potential impact to services. Further updates will be provided at 14:15
We will update here further as soon as we know more.
12:30
CityFibre have advised they are still seeing issues in certain areas of the UK.
They have provided the following update:
Investigations into the issue are ongoing. Following internal technical calls, the resolver teams are currently focusing efforts on stabilising the impacted services. Once these have been stabilised, they can then focus on finding the root cause and a permanent fix. Another internal technical call is taking place at 12:30 and further updates will be provided following this at 13:15
We will update here further as soon as we know more.
11:30
CityFibre have advised they are still seeing issues in certain areas of the UK.
They have provided the following update:
NOC and architecture & engineering teams are currently working with external vendor support to find the root cause and restore services. Further updates will be provided in the next hour.
We will update further as soon as we know more.
10:30
CityFibre have advised they are still seeing issues in certain areas of the UK but currently we are not seeing this affect any of our customers.
They have provided the following update:
We are currently seeing a recurrence of the issues following the work earlier this morning. Senior network engineers are continuing to investigate the root cause and further updates will be provided as soon as possible.
High level investigations into the issue are continuing to take place. At present, the problem will cause packet loss, latency or instances of total loss of service and is affecting multiple tunnels across the CityFibre network.
We will update further as soon as we know more.
07:30
We can see many services for customers in Peterborough appear to be back online and stable this morning. Most services appear to have come back online around 03:00.
CityFibre have provided the following update:
Following a re-boot of a core device on-site, services have begun to restore across the network for Peterborough connections.
We now advise to ensure that any services still down, are fully re-booted, and should the issue persist, please contact the service desk to advise the remaining connection issues.
A full report of the root cause is still currently ongoing and to be provided at a later time.
If your service has not been restored as of yet please reboot your CityFibre ONT and any related CPE in the first instance. If issues still persist please contact our Support Team via usual channels.
23:00
CityFibre have confirmed they are aware of an MSO (Major Service Outage) in Peterborough affecting a large number of customers and they have dispatched engineers to investigate further. Further updates to follow.
We will update here when we know more.
22:25
We have had alerts on our monitoring of several CityFibre circuits going offline in Peterborough in the last 30 minutes.
This issue has been raised with CityFibre to investigating further.
We will update here when we know more.
17:15
CityFibre have advised this MSO (Major Service Outage) also covers Sheffield and Nottingham in addition to Peterborough.
We are now seeing good stability of connections that have come back online and have received a further update from CityFibre to advise that this issue should now be resolved.
Services have been restored. A reboot was required which has resulted in the data necessary to carry out further root cause analysis to be lost. A reboot in this instance was the last available option in order for us to restore service.
Due to this we will continue to monitor for the next 48hours.
If service has not been restored as of yet please reboot CityFibre ONT and any related CPE and get in touch if still experiencing issues.
We are not currently expecting further related issues at this time. Please monitor and advise if further issues occur via usual channels.
16:45
CityFibre have advised this MSO (Major Service Outage) also covers Sheffield and Nottingham in addition to Peterborough.
We are starting to see some connections come back online for some customers now but have not received an all clear as of yet from CityFibre or a further update.
Last we heard just prior to this the engineer was attempting to restore connectivity to the affected Core Routers.
Until we hear further from CityFibre connectivity is still considered ‘at risk’ for the time being and further drops may occur.
14:30
CityFibre have advised this MSO (Major Service Outage) also covers Sheffield and Nottingham in addition to Peterborough.
Please see further update below:
Situation:
Services connected to our Peterborough POP seen to be down
Action:
Engineers attending to investigate further
Next Step(s):
Engineers are expected to arrive on site at around 1500 to carry out investigations.
Expectation:
Further update to follow once engineer has attended and assessed the situation.
13:00
CityFibre have advised this MSO (Major Service Outage) also covers Sheffield and Nottingham in addition to Peterborough.
We have received a brief update from them a few minutes ago:
Sheffield and Nottingham core routers are back online but we still have an issue in Peterborough. Engineers are being dispatched.
12:00
We are aware of an outage in Peterborough affecting CityFibre customers.
CityFibre are aware of the issue and are investigating. As soon as we know more we will update here and anyone who has raised a support request with us.
17:00
Microsoft Support now believe this issue is resolved. Please see the full update below:
User Impact: Users may have been unable to access the Exchange Online service through the Outlook client.
More info: Additionally, some users may have also experienced impact connecting through Outlook on the web.
Final status: We’ve determined that a subset of components, responsible for directing user requests, received traffic whilst it was going through an update and was not configured correctly to process the traffic. We’ve routed connections to alternate components and confirmed that the issue is remediated after monitoring the service.
Scope of impact: Customers connecting to the Exchange Online service through our Europe and United Kingdom data centers may have experienced impact.
Preliminary root cause: A small section of front end components, used to direct user requests, was inadvertently sent traffic whilst it was receiving an update and was not correctly configured to process requests, which resulted in impact.
We are seeing service return to normal for our users. If you are still experiencing issues please get in touch through the usual support channels.
15:45
We have received a further update from Microsoft Support on this issue. Please see below:
User Impact: Users may be unable to access the Exchange Online service through the Outlook client.
More info: While we are focused on remediation, users who have access, may be able to use Outlook on the web and Outlook mobile apps to connect the service.
Current status: We’ve identified an issue where users from Europe and United Kingdom are unable to connect to Exchange Online through the Outlook client. Our telemetry indicates that the service has now recovered but we are investigating the underlying cause and monitoring the service to ensure full recovery.
Scope of impact: Customers connecting to the Exchange Online service through our Europe and United Kingdom data centers may experience impact.
Accessing Exchange Online via other means such as Outlook for Web or a Mobile Device appear to be unaffected at this time.
15:00
We are aware of issues intermittently affecting users of Microsoft 365 to access Exchange Online (Email) via the Outlook Client.
Microsoft are already aware of the issue and are working on. Please see below update from Microsoft:
User Impact: Users may be unable to connect to the Exchange Online service using the Outlook client.
Current status: We’re investigating a potential issue.
Accessing Exchange Online via other means such as Outlook for Web or a Mobile Device appear to be unaffected at this time.
21:00
Microsoft Support now believe this issue is resolved. Please see the full update below:
User Impact: Users may have been unable to connect to the Exchange Online service using the Outlook client.
Final status: We’ve identified a Domain Name System (DNS) issue within components used to process the Outlook client connectivity to the service. This issue has self-recovered and after an extensive period of monitoring, we’re declaring the issue resolved.
Scope of impact: Impact was specific to users located in Europe.
Preliminary root cause: A DNS issue within front-end components, used to direct user requests, resulted in the Outlook client disconnecting from the service.
We are seeing service return to normal for our users. If you are still experiencing issues please get in touch through the usual support channels.
16:00
We have received a further update from Microsoft Support on this issue. Please see below:
User Impact: Users may be unable to connect to the Exchange Online service using the Outlook client.
Current status: We’ve identified a potential networking issue and we’re analyzing diagnostic data to determine the next troubleshooting steps.
Scope of impact: Impact is specific to users located in Europe
Accessing Exchange Online via other means such as Outlook for Web or a Mobile Device appear to be unaffected at this time.
15:00
We are aware of issues affecting users of Microsoft 365 to access Exchange Online (Email) via the Outlook Client.
Microsoft are already aware of the issue and are working on. Please see below update from Microsoft:
User Impact: Users may be unable to connect to the Exchange Online service using the Outlook client.
Current status: We’re reviewing system telemetry to isolate the source of the issue.
Scope of impact: Impact is limited to users located in Europe.
Accessing Exchange Online via other means such as Outlook for Web or a Mobile Device appear to be unaffected at this time.
12:00
TalkTalk have provided the following update:
“We are pleased to confirm that the earlier problem has now been resolved and access has been restored, once again we apologise for any inconvenience caused.”
If you are experiencing issues with your business broadband please get in touch – 01733830555
11:30
We have been made aware of issues with TalkTalk broadband across the UK.
At this time this seems to largely be confined to residential but we are monitoring with our suppliers.
Currently we do not believe any of our customers are affected.
If you are experiencing issues with your business broadband please get in touch – 01733830555
We are aware of issues with the O2 Network affecting some customers across the UK.
Please see O2’s Website for further information: https://status.o2.co.uk
17:30
Services should now have returned to normal.
The East coast Fibre restoration works have been completed and full resiliency across the supplier’s network has been restored.
All services should now be functioning as expected. Please contact the support team via the usual channels if you have any ongoing issues.
Thank you for your continued patience.
09:30
Services should now be starting to return to normal. Latest update from the supplier is below:
“The fibre issue in the London area has now been resolved as of 05:01, our monitoring and testing has confirmed that service is restoring across our platforms and we will continue to monitor network stability throughout this morning. We apologise for any disruption this incident has caused to your service. “
We have seen many customers services return to stability this morning and will continue to monitor this issue.
00:30
Issues are still ongoing. Latest update from the supplier is below:
“Work on the stabilisation of re-routed traffic across the temporary network connection remains a challenge and we have engaged our technology partner to work alongside us on this. This will continue to be a priority throughout the evening, but unfortunately we can’t yet provide an outline timescale on when this will complete.
Fibre engineers have identified the break between two fibre joints in the London area and are investigating the options to replace this with a local spare. Once we know the availability of a spare fibre we hope to be able to provide an estimated time to restore.”
The supplier’s engineers will be continuing to work on this throughout the night and will look to provide a full assessment and update at 08:00. They will still look to provide any material changes in status as they happen overnight but are looking to provide a full update across all areas at 08:00.
Until at least one of the Fibre breaks is restored service disruption is expected to continue.
Further updates are expected by 08:00
21:30
Issues are still ongoing. Latest update from the supplier is below:
“Fibre engineers are now on site and beginning work on the restoration. Our network team have been continuing the work on the network re-routing of voice and connectivity services and we expect to restore those services by 10pm tonight. We’re confident the re-routing work will fully restore services, however services remain at risk until the fibre restoration work is completed.“
Until at least one of the Fibre breaks is restored service disruption is expected to continue.
Further updates are expected by 23:00
20:45
Issues are still ongoing. Latest update from the supplier is below:
“We have isolated the fault to a dual fibre break on the East coast and the London area of our national network. Our technology partner has been engaged and they are deploying engineers to address both breaks. The incident continues to impact our voice and connectivity services and we are taking mitigating action at a network level to try to restore connectivity where we can.
We estimate engineers to arrive at the fibre break in the London area at 9pm at which point they can assess the extent of the work required to restore the fibre connectivity. Work will begin at 9pm on fibre break within the Leicester area, however we know the restoration work here is extensive due to an electrical incident in that area.”
Until at least one of the Fibre breaks is restored service disruption is expected to continue.
Further updates are expected by 22:00
17:15
The issues with Broadband and Phone lines for some customers are still ongoing.
The issues have been identified by the supplier to be down to a major fibre break in the supplier’s network infrastructure which is then causing large amounts of congestion across the network affecting Voice and Data connectivity.
Engineers are working to identify the cause of the break and repair it. In the meantime they are also working to route affected traffic over alternative routes to alleviate congestion and restore service.
Next update is expected by 18:30
16:30
The issues with Broadband and Phone lines for some customers are still ongoing.
The issues are still being investigated by the supplier’s engineers.
We are expecting a further update by 17:30
15:00
We are aware of issues across the country affecting Broadband and Phones for some customers.
Issues range from the following:
We are investigating these issues with the supplier and will update when we know more.
Further Update from Microsoft:
Current status: After further review of the captured data, we’ve determined that a recent authentication change has introduced a problem where users with multiple accounts are unable to authenticate to the service properly, preventing them from accessing their mailboxes. We’re working to develop a fix to address this issue and will provide an estimated timeline for mitigation early next week.
Scope of impact: Users may experience this issue if they operate multiple related and trusted Enterprise log-in accounts across Microsoft 365 and Windows.
Root cause: A recent authentication change has introduced a problem where users with multiple accounts are unable to authenticate to the service properly, preventing them from accessing their mailboxes.
Microsoft have also advised the following workaround may restore access to Outlook in the interim:
1) Close Outlook and other Office applications.
2) Open an Office desktop application, such as Word, PowerPoint or Excel.
3) Navigate to File > Account > Switch accounts, and then click the ellipsis next to the account that’s not having the problem.
4) Click Sign out and then OK. *Note: Users that have multiple accounts may find it difficult to determine which account to sign out of; though, they should be able to sign back in later if there is an issue with this process.
5) Close the Office application used for this process and re-start the Outlook desktop client.
6) When prompted for a password, click “sign in to a different account” and enter the correct information.
If the above does not work users will need to revert to using Webmail as previously advised.
We are aware of an issue affecting Users on Office 365 ability to use Outlook to send and receive email. Please see details from Microsoft below:
User Impact: Users with multiple enterprise sign-in accounts can’t connect to Exchange Online via the Outlook desktop client. Users experiencing this problem will see the Outlook desktop client showing as disconnected and may receive the following error when attempting to send mail: “Sending and receiving reported error (0x80040115). The connection to the Microsoft Exchange Server is unavailable. Outlook must be online or connected to complete this action.”
Users may be able to leverage one of the following options to circumvent impact for this event:
– Affected users may be able to use Outlook on the web or mobile clients to access their mailboxes.
If you are being affected by this issue please attempt to use Outlook Webmail for the time being until Microsoft resolve the issue.
To access Outlook for the Web browse to https://outlook.office365.com and login with your Office 365 Email Address/Username and Password.
06:20
2G, 3G, and 4G data back online.
If customers are still experiencing issues reconnecting to mobile data, please advise them to turn their handsets off and on again (or turn flight mode on and then off). This will refresh the service and trigger it to actively search for a data signal.
Please advise any customers experiencing slow or intermittent data access that this may be down to the large number of customers reconnecting to data at the same time.
Thank you for your patience and understanding.
08:37
The O2 network is experiencing an issue with DATA across the UK.
We apologise for any inconvenience and will share updates with you as soon as we have them.
Thank you for your patience and understanding.
10:15
We are aware of some data centre issues which may cause customers to have difficulty connecting. We are currently working on the issue.
15:12 pm
The supplier has now confirmed the issue has been resolved.
14:23 pm
We have been informed that the issue with slowness has now been resolved. We will continue to monitor the issue over time.
13:26pm
The supplier has identified the issue and is currently working on the resolution. We will continue to push updates once we have more information.
13:06pm
The supplier is currently investigating the slowness issues, we will update this when we have more information.
10:33am
As per our previous update power is restored and we believe all services are backup, if you experience further issues please contact our support desk.
08:59
We have been working on an issue in our data centre affecting cloud/hosted platforms. Power has been restored and we are working to get services back online as quickly as possible. Further updates within 2 hours.
12:52
The issue affecting some customers broadband services has now been resolved. If you still experience issues please contact the support team on 01733 667755.
10:02
The fault affecting customers has now been identified and is being worked on, we are hoping that services will be restored soon.
09:09
We are aware of a national broadband outage which is affecting some customers. The issue is being worked on, we are keeping up to date with developments and will update here within the next 2 hours.
09:25
We are aware of an issue on the iExchange platform which is causing a small number of customers to experience email issues, we are working on this issue as our top priority, we will update here..
|
10:18
The issue affecting connectivity for some customers is now resolved and all affected customer are now working normally.
09:00
We are aware of a connectivity issue affecting some of our customers, it is being investigate by our supplier and we will update as soon as possible. They have engineers working on it.
Symptons: Loss of internet and voice traffic
10:49
One of our third party hosted mail platforms is currently experiencing an outage which may result in some users not being able to send or receive email. This is being investigated as a priority, we will update progress as we receive it.
10:27
The Hosted Phone outage is now resolved and service should be restored to all customers. We are working on removing all diverts, this may take some time due to the volume.
08:52
We are aware of some issues with some customers hosted phone systems. Engineers are working on this in the data centre and we will provide an update here as soon as we have it.
If you require a divert please email support@greencityict.co.uk with the subject Hosted Divert, please include the number you wish to divert to.
14:34
We are aware of some issues with Microsoft Outlook which have been ongoing for some time. Microsoft has confirmed that some users are unable to send email or to access their accounts.
The issue appears to be intermittent and affecting some customers but no means all.
We will continue to monitor the situation and update this page as soon as we know more.
As this is an intermittent issue, if you are experiencing issues with Outlook please contact us by telephone on 01733 667755 so that we can investigate.
15:40
This issue has been investigated and resolved. There are a very small number of customers being affected, this customers are being support by our Engineers and Technicians.
08:58
We are aware of a broadband outage affecting some customers which we are currently investigating. Any updates will be posted here as soon as we have them.
The earlier issue was caused by a main network operation centre which suffered a power cut as of 9:13am due to external works in the local area. This also affected our ability to receive calls. Power is currently being provided via a back up service and recovery was from approximately 9:30am. There will have been some service interruption to voice and data services for a number of customers if their service was primarily hosted in this location, while services were rerouted to our resilient locations, and a number of services may still be recovering.
09:52
We are aware of a widespread telecoms issue and we are investigating. This issue is affecting us so please email you support request to support@greencityict.co.uk. Further updates will be available here as soon as we have them.
11:25
The MyDesktop Platform issue has now been resolved, if you are still experiencing issue please let us know in the usual way.
11:00
The issue affecting the platorm is still being worked on, further updates on this issue will be posted here
09:43
We are still working on the issue affected MyDesktop Platform customers, we will provide another update soon.
09:05
We are aware of an issue on the MyDesktop Platform this morning which is displaying as a certificate error.
The error appears to be preventing people being able to logon and connect to the system.
This is currently being worked on, we will provide a further update at 9:45am
Further updates can be found here.
14:43 – Resolved
The issue has been identified and resolved. We are currently contacting affected customers to ensure they are no longer experiencing issues.
The suppliers are aware of this issue and continue to work hard to resolve it.
11:43
We are currently aware of a call quality issue affecting some customers who have SpliceCom Hosted Phone Systems this is affecting external calls. We have notified SpliceCom Support who are investigating further with their carrier and are working to resolve the issue.
We apologise for any inconvenience caused. Please check back here for updates.
11:12
The issue regarding a national broadband outage appears to be resolved although the supplier fault has not yet been closed. Our customers are seeing services being restored, the restoration of services started approximately half an hour ago. Due to this being a national outage we may not receive an outage report, if we do it will be posted here.
09:39
We are aware of a major national broadband outage affecting businesses all over the country. We are working with our suppliers to resolve this issue as quickly as possible and will post updates here as often as possible.
If you require diverts on your phones please contact our help desk on 01733 667755.
13:18
The certificate issue on the mail platform has now been resolved.
The affected users Outlook clients should now be connecting and incoming and outgoing emails should be flowing (although this may take a little time). If you were an affected customer still experiencing issues with Outlook connecting close Outlook and re-open. If the issue persists please contact us on 01733 667755.
12:25
We are currently making progress on this issue and we believe mail to mobile devices has now been restored. We are continuing to work towards a full resolution for Outlook connectivity
11:02
We are still working on the issue affecting some customers using Outlook and hope to have this resolved very soon.
As previously mentioned Webmail is working for both sending and receiving emails using the following address https://mail.c1.questcloud.net/owa
Once again we apologise for the invconvenience
10:00
We are working on an issue with a corrupt certificate on one of our mail platforms which is causing Outlook not to connect for some of our customers. Webmail is still working by clicking through the certificate/security warnings.
We will post another update as soon as possible
Webmail address: https://mail.c1.questcloud.net/owa
We apologise for the inconvenience.
08:55
We are currently aware of a certificate error on part of our email platform that is preventing some users Outlook from being able to connect. Our engineers are currently working on resolving the issue at the moment. This issue only affects users connecting on Outlook, the Web mail portal is working without issue and users can send and receive email on that portal.
The Web mail address is https://mail.c1.questcloud.net/owa
Please check back here for updates
12:45
Apologies for the late notice but we have just received the notice below from our cloud service provider. There will be emergency maintenance late tonight to perform essential maintenance.
If you have any questions related to this please contact our helpdesk.
Effected systems: Virtual Private Cloud, questMail, VPS, questCloud Virtual Desktop
Description of works: We are currently experiencing an issue across our primary Hyper-V Clusters with Windows Remote Management Instrumentation (WinRM). This is effecting our ability to communicate with our Host Clusters and provide management access to customers. Some of the primary tasks WinRM provides are Virtual Machine load balancing across the clusters, Virtual Machine Migration, console management and storage management.
After consulting with the manufacturer of the software Microsoft we have been advised that the next course of action is to perform restarts of every node in our Host Clusters. Under normal operations this would not be an issue however, one of the effected systems of this fault is our ability to “Live Migrate” Customer Virtual Machines from one Host to another.
As a result of this completing this maintenance will effect service while each node is restarted.
Impact Details: Each Cluster Node will have its Virtual Machine guests placed into a “saved” or “paused” state then the node will be restarted and the guest Virtual machines resumed. During this time the Virtual machine will be unavailable, once the Virtual Machine is resumed it will continue from where it was before maintenance began without being restarted.
Service will be removed at 22:00 this evening and we are forecasting a restore of all services by 02:00, however it may be necessary to extend this until 04:00 in a worst case scenario.
We have been informed by City Fibre that between 06:00 and 15:00 on Sunday 27/11/16 Peterborough City Council are committed to completing the upgrade of their power distribution in the Town Hall. There should be no disruption of service, however due to the nature of the maintenance services are deemed “at risk” for the duration of this period. In the event of an outage and your service not automatically being restored you may need to restart both the white ONT & the supplied router or own equipment.
.
On Friday 4th November our office will be closing at 4pm.
Following the completion of our recent acquisition of Total Solutions and our continued dedication to providing the best service through a motivated and knowledgeable team we will be closing early on Friday 4th November 2016 for some team building.
It is important that all team members attend so from 4pm on Friday 4th November if you have an urgent support or maintenance issue please call in the usual way and follow the prompts, you will be asked to leave a message and one of our technical team will call you back shortly after the end of our meeting at 5pm.
If you have any other queries including non-urgent support queries from 4pm on Friday please email the relevant team or call on 01733 667755, leave a message and we will respond on Monday 7th November 2016.
If you have any questions please email getintouch@greencityict.co.uk
14.32 – The outgoing call issues have now been resolved.
We are aware of an issue affecting some customers when making outbound calls. When dialing out to some numbers some customers are receiving a number unobtainable tone. We are currently working with the carrier to resolve the issue as quickly as possible.