17:00
Microsoft Support now believe this issue is resolved. Please see the full update below:
User Impact: Users may have been unable to access the Exchange Online service through the Outlook client.
More info: Additionally, some users may have also experienced impact connecting through Outlook on the web.
Final status: We’ve determined that a subset of components, responsible for directing user requests, received traffic whilst it was going through an update and was not configured correctly to process the traffic. We’ve routed connections to alternate components and confirmed that the issue is remediated after monitoring the service.
Scope of impact: Customers connecting to the Exchange Online service through our Europe and United Kingdom data centers may have experienced impact.
Preliminary root cause: A small section of front end components, used to direct user requests, was inadvertently sent traffic whilst it was receiving an update and was not correctly configured to process requests, which resulted in impact.
We are seeing service return to normal for our users. If you are still experiencing issues please get in touch through the usual support channels.
15:45
We have received a further update from Microsoft Support on this issue. Please see below:
User Impact: Users may be unable to access the Exchange Online service through the Outlook client.
More info: While we are focused on remediation, users who have access, may be able to use Outlook on the web and Outlook mobile apps to connect the service.
Current status: We’ve identified an issue where users from Europe and United Kingdom are unable to connect to Exchange Online through the Outlook client. Our telemetry indicates that the service has now recovered but we are investigating the underlying cause and monitoring the service to ensure full recovery.
Scope of impact: Customers connecting to the Exchange Online service through our Europe and United Kingdom data centers may experience impact.
Accessing Exchange Online via other means such as Outlook for Web or a Mobile Device appear to be unaffected at this time.
15:00
We are aware of issues intermittently affecting users of Microsoft 365 to access Exchange Online (Email) via the Outlook Client.
Microsoft are already aware of the issue and are working on. Please see below update from Microsoft:
User Impact: Users may be unable to connect to the Exchange Online service using the Outlook client.
Current status: We’re investigating a potential issue.
Accessing Exchange Online via other means such as Outlook for Web or a Mobile Device appear to be unaffected at this time.
21:00
Microsoft Support now believe this issue is resolved. Please see the full update below:
User Impact: Users may have been unable to connect to the Exchange Online service using the Outlook client.
Final status: We’ve identified a Domain Name System (DNS) issue within components used to process the Outlook client connectivity to the service. This issue has self-recovered and after an extensive period of monitoring, we’re declaring the issue resolved.
Scope of impact: Impact was specific to users located in Europe.
Preliminary root cause: A DNS issue within front-end components, used to direct user requests, resulted in the Outlook client disconnecting from the service.
We are seeing service return to normal for our users. If you are still experiencing issues please get in touch through the usual support channels.
16:00
We have received a further update from Microsoft Support on this issue. Please see below:
User Impact: Users may be unable to connect to the Exchange Online service using the Outlook client.
Current status: We’ve identified a potential networking issue and we’re analyzing diagnostic data to determine the next troubleshooting steps.
Scope of impact: Impact is specific to users located in Europe
Accessing Exchange Online via other means such as Outlook for Web or a Mobile Device appear to be unaffected at this time.
15:00
We are aware of issues affecting users of Microsoft 365 to access Exchange Online (Email) via the Outlook Client.
Microsoft are already aware of the issue and are working on. Please see below update from Microsoft:
User Impact: Users may be unable to connect to the Exchange Online service using the Outlook client.
Current status: We’re reviewing system telemetry to isolate the source of the issue.
Scope of impact: Impact is limited to users located in Europe.
Accessing Exchange Online via other means such as Outlook for Web or a Mobile Device appear to be unaffected at this time.
12:00
TalkTalk have provided the following update:
“We are pleased to confirm that the earlier problem has now been resolved and access has been restored, once again we apologise for any inconvenience caused.”
If you are experiencing issues with your business broadband please get in touch – 01733830555
11:30
We have been made aware of issues with TalkTalk broadband across the UK.
At this time this seems to largely be confined to residential but we are monitoring with our suppliers.
Currently we do not believe any of our customers are affected.
If you are experiencing issues with your business broadband please get in touch – 01733830555
We are aware of issues with the O2 Network affecting some customers across the UK.
Please see O2’s Website for further information: https://status.o2.co.uk
17:30
Services should now have returned to normal.
The East coast Fibre restoration works have been completed and full resiliency across the supplier’s network has been restored.
All services should now be functioning as expected. Please contact the support team via the usual channels if you have any ongoing issues.
Thank you for your continued patience.
09:30
Services should now be starting to return to normal. Latest update from the supplier is below:
“The fibre issue in the London area has now been resolved as of 05:01, our monitoring and testing has confirmed that service is restoring across our platforms and we will continue to monitor network stability throughout this morning. We apologise for any disruption this incident has caused to your service. “
We have seen many customers services return to stability this morning and will continue to monitor this issue.
00:30
Issues are still ongoing. Latest update from the supplier is below:
“Work on the stabilisation of re-routed traffic across the temporary network connection remains a challenge and we have engaged our technology partner to work alongside us on this. This will continue to be a priority throughout the evening, but unfortunately we can’t yet provide an outline timescale on when this will complete.
Fibre engineers have identified the break between two fibre joints in the London area and are investigating the options to replace this with a local spare. Once we know the availability of a spare fibre we hope to be able to provide an estimated time to restore.”
The supplier’s engineers will be continuing to work on this throughout the night and will look to provide a full assessment and update at 08:00. They will still look to provide any material changes in status as they happen overnight but are looking to provide a full update across all areas at 08:00.
Until at least one of the Fibre breaks is restored service disruption is expected to continue.
Further updates are expected by 08:00
21:30
Issues are still ongoing. Latest update from the supplier is below:
“Fibre engineers are now on site and beginning work on the restoration. Our network team have been continuing the work on the network re-routing of voice and connectivity services and we expect to restore those services by 10pm tonight. We’re confident the re-routing work will fully restore services, however services remain at risk until the fibre restoration work is completed.“
Until at least one of the Fibre breaks is restored service disruption is expected to continue.
Further updates are expected by 23:00
20:45
Issues are still ongoing. Latest update from the supplier is below:
“We have isolated the fault to a dual fibre break on the East coast and the London area of our national network. Our technology partner has been engaged and they are deploying engineers to address both breaks. The incident continues to impact our voice and connectivity services and we are taking mitigating action at a network level to try to restore connectivity where we can.
We estimate engineers to arrive at the fibre break in the London area at 9pm at which point they can assess the extent of the work required to restore the fibre connectivity. Work will begin at 9pm on fibre break within the Leicester area, however we know the restoration work here is extensive due to an electrical incident in that area.”
Until at least one of the Fibre breaks is restored service disruption is expected to continue.
Further updates are expected by 22:00
17:15
The issues with Broadband and Phone lines for some customers are still ongoing.
The issues have been identified by the supplier to be down to a major fibre break in the supplier’s network infrastructure which is then causing large amounts of congestion across the network affecting Voice and Data connectivity.
Engineers are working to identify the cause of the break and repair it. In the meantime they are also working to route affected traffic over alternative routes to alleviate congestion and restore service.
Next update is expected by 18:30
16:30
The issues with Broadband and Phone lines for some customers are still ongoing.
The issues are still being investigated by the supplier’s engineers.
We are expecting a further update by 17:30
15:00
We are aware of issues across the country affecting Broadband and Phones for some customers.
Issues range from the following:
We are investigating these issues with the supplier and will update when we know more.
Further Update from Microsoft:
Current status: After further review of the captured data, we’ve determined that a recent authentication change has introduced a problem where users with multiple accounts are unable to authenticate to the service properly, preventing them from accessing their mailboxes. We’re working to develop a fix to address this issue and will provide an estimated timeline for mitigation early next week.
Scope of impact: Users may experience this issue if they operate multiple related and trusted Enterprise log-in accounts across Microsoft 365 and Windows.
Root cause: A recent authentication change has introduced a problem where users with multiple accounts are unable to authenticate to the service properly, preventing them from accessing their mailboxes.
Microsoft have also advised the following workaround may restore access to Outlook in the interim:
1) Close Outlook and other Office applications.
2) Open an Office desktop application, such as Word, PowerPoint or Excel.
3) Navigate to File > Account > Switch accounts, and then click the ellipsis next to the account that’s not having the problem.
4) Click Sign out and then OK. *Note: Users that have multiple accounts may find it difficult to determine which account to sign out of; though, they should be able to sign back in later if there is an issue with this process.
5) Close the Office application used for this process and re-start the Outlook desktop client.
6) When prompted for a password, click “sign in to a different account” and enter the correct information.
If the above does not work users will need to revert to using Webmail as previously advised.
We are aware of an issue affecting Users on Office 365 ability to use Outlook to send and receive email. Please see details from Microsoft below:
User Impact: Users with multiple enterprise sign-in accounts can’t connect to Exchange Online via the Outlook desktop client. Users experiencing this problem will see the Outlook desktop client showing as disconnected and may receive the following error when attempting to send mail: “Sending and receiving reported error (0x80040115). The connection to the Microsoft Exchange Server is unavailable. Outlook must be online or connected to complete this action.”
Users may be able to leverage one of the following options to circumvent impact for this event:
– Affected users may be able to use Outlook on the web or mobile clients to access their mailboxes.
If you are being affected by this issue please attempt to use Outlook Webmail for the time being until Microsoft resolve the issue.
To access Outlook for the Web browse to https://outlook.office365.com and login with your Office 365 Email Address/Username and Password.
06:20
2G, 3G, and 4G data back online.
If customers are still experiencing issues reconnecting to mobile data, please advise them to turn their handsets off and on again (or turn flight mode on and then off). This will refresh the service and trigger it to actively search for a data signal.
Please advise any customers experiencing slow or intermittent data access that this may be down to the large number of customers reconnecting to data at the same time.
Thank you for your patience and understanding.
08:37
The O2 network is experiencing an issue with DATA across the UK.
We apologise for any inconvenience and will share updates with you as soon as we have them.
Thank you for your patience and understanding.
10:15
We are aware of some data centre issues which may cause customers to have difficulty connecting. We are currently working on the issue.
15:12 pm
The supplier has now confirmed the issue has been resolved.
14:23 pm
We have been informed that the issue with slowness has now been resolved. We will continue to monitor the issue over time.
13:26pm
The supplier has identified the issue and is currently working on the resolution. We will continue to push updates once we have more information.
13:06pm
The supplier is currently investigating the slowness issues, we will update this when we have more information.
10:33am
As per our previous update power is restored and we believe all services are backup, if you experience further issues please contact our support desk.
08:59
We have been working on an issue in our data centre affecting cloud/hosted platforms. Power has been restored and we are working to get services back online as quickly as possible. Further updates within 2 hours.
12:52
The issue affecting some customers broadband services has now been resolved. If you still experience issues please contact the support team on 01733 667755.
10:02
The fault affecting customers has now been identified and is being worked on, we are hoping that services will be restored soon.
09:09
We are aware of a national broadband outage which is affecting some customers. The issue is being worked on, we are keeping up to date with developments and will update here within the next 2 hours.
09:25
We are aware of an issue on the iExchange platform which is causing a small number of customers to experience email issues, we are working on this issue as our top priority, we will update here..
|
10:18
The issue affecting connectivity for some customers is now resolved and all affected customer are now working normally.
09:00
We are aware of a connectivity issue affecting some of our customers, it is being investigate by our supplier and we will update as soon as possible. They have engineers working on it.
Symptons: Loss of internet and voice traffic
10:49
One of our third party hosted mail platforms is currently experiencing an outage which may result in some users not being able to send or receive email. This is being investigated as a priority, we will update progress as we receive it.
10:27
The Hosted Phone outage is now resolved and service should be restored to all customers. We are working on removing all diverts, this may take some time due to the volume.
08:52
We are aware of some issues with some customers hosted phone systems. Engineers are working on this in the data centre and we will provide an update here as soon as we have it.
If you require a divert please email support@greencityict.co.uk with the subject Hosted Divert, please include the number you wish to divert to.
14:34
We are aware of some issues with Microsoft Outlook which have been ongoing for some time. Microsoft has confirmed that some users are unable to send email or to access their accounts.
The issue appears to be intermittent and affecting some customers but no means all.
We will continue to monitor the situation and update this page as soon as we know more.
As this is an intermittent issue, if you are experiencing issues with Outlook please contact us by telephone on 01733 667755 so that we can investigate.
15:40
This issue has been investigated and resolved. There are a very small number of customers being affected, this customers are being support by our Engineers and Technicians.
08:58
We are aware of a broadband outage affecting some customers which we are currently investigating. Any updates will be posted here as soon as we have them.
The earlier issue was caused by a main network operation centre which suffered a power cut as of 9:13am due to external works in the local area. This also affected our ability to receive calls. Power is currently being provided via a back up service and recovery was from approximately 9:30am. There will have been some service interruption to voice and data services for a number of customers if their service was primarily hosted in this location, while services were rerouted to our resilient locations, and a number of services may still be recovering.
09:52
We are aware of a widespread telecoms issue and we are investigating. This issue is affecting us so please email you support request to support@greencityict.co.uk. Further updates will be available here as soon as we have them.
11:25
The MyDesktop Platform issue has now been resolved, if you are still experiencing issue please let us know in the usual way.
11:00
The issue affecting the platorm is still being worked on, further updates on this issue will be posted here
09:43
We are still working on the issue affected MyDesktop Platform customers, we will provide another update soon.
09:05
We are aware of an issue on the MyDesktop Platform this morning which is displaying as a certificate error.
The error appears to be preventing people being able to logon and connect to the system.
This is currently being worked on, we will provide a further update at 9:45am
Further updates can be found here.
14:43 – Resolved
The issue has been identified and resolved. We are currently contacting affected customers to ensure they are no longer experiencing issues.
The suppliers are aware of this issue and continue to work hard to resolve it.
11:43
We are currently aware of a call quality issue affecting some customers who have SpliceCom Hosted Phone Systems this is affecting external calls. We have notified SpliceCom Support who are investigating further with their carrier and are working to resolve the issue.
We apologise for any inconvenience caused. Please check back here for updates.
11:12
The issue regarding a national broadband outage appears to be resolved although the supplier fault has not yet been closed. Our customers are seeing services being restored, the restoration of services started approximately half an hour ago. Due to this being a national outage we may not receive an outage report, if we do it will be posted here.
09:39
We are aware of a major national broadband outage affecting businesses all over the country. We are working with our suppliers to resolve this issue as quickly as possible and will post updates here as often as possible.
If you require diverts on your phones please contact our help desk on 01733 667755.
13:18
The certificate issue on the mail platform has now been resolved.
The affected users Outlook clients should now be connecting and incoming and outgoing emails should be flowing (although this may take a little time). If you were an affected customer still experiencing issues with Outlook connecting close Outlook and re-open. If the issue persists please contact us on 01733 667755.
12:25
We are currently making progress on this issue and we believe mail to mobile devices has now been restored. We are continuing to work towards a full resolution for Outlook connectivity
11:02
We are still working on the issue affecting some customers using Outlook and hope to have this resolved very soon.
As previously mentioned Webmail is working for both sending and receiving emails using the following address https://mail.c1.questcloud.net/owa
Once again we apologise for the invconvenience
10:00
We are working on an issue with a corrupt certificate on one of our mail platforms which is causing Outlook not to connect for some of our customers. Webmail is still working by clicking through the certificate/security warnings.
We will post another update as soon as possible
Webmail address: https://mail.c1.questcloud.net/owa
We apologise for the inconvenience.
08:55
We are currently aware of a certificate error on part of our email platform that is preventing some users Outlook from being able to connect. Our engineers are currently working on resolving the issue at the moment. This issue only affects users connecting on Outlook, the Web mail portal is working without issue and users can send and receive email on that portal.
The Web mail address is https://mail.c1.questcloud.net/owa
Please check back here for updates
12:45
Apologies for the late notice but we have just received the notice below from our cloud service provider. There will be emergency maintenance late tonight to perform essential maintenance.
If you have any questions related to this please contact our helpdesk.
Effected systems: Virtual Private Cloud, questMail, VPS, questCloud Virtual Desktop
Description of works: We are currently experiencing an issue across our primary Hyper-V Clusters with Windows Remote Management Instrumentation (WinRM). This is effecting our ability to communicate with our Host Clusters and provide management access to customers. Some of the primary tasks WinRM provides are Virtual Machine load balancing across the clusters, Virtual Machine Migration, console management and storage management.
After consulting with the manufacturer of the software Microsoft we have been advised that the next course of action is to perform restarts of every node in our Host Clusters. Under normal operations this would not be an issue however, one of the effected systems of this fault is our ability to “Live Migrate” Customer Virtual Machines from one Host to another.
As a result of this completing this maintenance will effect service while each node is restarted.
Impact Details: Each Cluster Node will have its Virtual Machine guests placed into a “saved” or “paused” state then the node will be restarted and the guest Virtual machines resumed. During this time the Virtual machine will be unavailable, once the Virtual Machine is resumed it will continue from where it was before maintenance began without being restarted.
Service will be removed at 22:00 this evening and we are forecasting a restore of all services by 02:00, however it may be necessary to extend this until 04:00 in a worst case scenario.
We have been informed by City Fibre that between 06:00 and 15:00 on Sunday 27/11/16 Peterborough City Council are committed to completing the upgrade of their power distribution in the Town Hall. There should be no disruption of service, however due to the nature of the maintenance services are deemed “at risk” for the duration of this period. In the event of an outage and your service not automatically being restored you may need to restart both the white ONT & the supplied router or own equipment.
.
On Friday 4th November our office will be closing at 4pm.
Following the completion of our recent acquisition of Total Solutions and our continued dedication to providing the best service through a motivated and knowledgeable team we will be closing early on Friday 4th November 2016 for some team building.
It is important that all team members attend so from 4pm on Friday 4th November if you have an urgent support or maintenance issue please call in the usual way and follow the prompts, you will be asked to leave a message and one of our technical team will call you back shortly after the end of our meeting at 5pm.
If you have any other queries including non-urgent support queries from 4pm on Friday please email the relevant team or call on 01733 667755, leave a message and we will respond on Monday 7th November 2016.
If you have any questions please email getintouch@greencityict.co.uk
14.32 – The outgoing call issues have now been resolved.
We are aware of an issue affecting some customers when making outbound calls. When dialing out to some numbers some customers are receiving a number unobtainable tone. We are currently working with the carrier to resolve the issue as quickly as possible.