tag:servcomm1.statuspage.io,2005:/historyServcomm Status - Incident History2024-03-20T16:17:17-07:00Servcommtag:servcomm1.statuspage.io,2005:Incident/203120622024-03-20T16:17:17-07:002024-03-20T16:17:17-07:00Wireless Carrier Connectivity Issues - Canada Wide<p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>16:17</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>12:36</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>11:35</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>10:55</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>10:14</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>09:51</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:servcomm1.statuspage.io,2005:Incident/200993322024-02-28T21:34:03-08:002024-02-28T21:34:03-08:00Intermittent Call Issues<p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>21:34</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>14:24</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>12:33</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented by our local carrier partner as of 1:59pm EST and we are monitoring the results.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>10:47</var> PST</small><br><strong>Update</strong> - The issue has been identified, our local carrier is experiencing a network connectivity issue. We are working with them at this time to determine cause and time to restoration. The Issue appears to be affecting calls to/from Telus & Rogers networks for the most part.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>10:10</var> PST</small><br><strong>Identified</strong> - The issue has been identified, our local carrier is experiencing a network connectivity issue. We are working with them at this time to determine cause and time to restoration.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>10:08</var> PST</small><br><strong>Monitoring</strong> - The issue has been identified, our local carrier is experiencing a network connectivity issue. We are working with them at this time to determine cause and time to restoration.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>10:03</var> PST</small><br><strong>Identified</strong> - The issue has been identified, our local carrier is experiencing a network connectivity issue. We are working with them at this time to determine cause and time to restoration.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>09:31</var> PST</small><br><strong>Investigating</strong> - We are currently investigating an apparent Network issues with some calls not completing, failing or having intermittent audio. We are working with our networks to find the cause and a resolution as soon as possible and will make updates as soon as we have further information</p>tag:servcomm1.statuspage.io,2005:Incident/200882642024-02-27T09:34:24-08:002024-02-27T09:34:24-08:003CX Service issue - partial outage<p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>09:34</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>08:30</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>08:28</var> PST</small><br><strong>Monitoring</strong> - A Fix is in place, all services are now back online, we will continue to monitor</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>07:37</var> PST</small><br><strong>Identified</strong> - The Issue has been identified, servers affected are being transferred to our failover service now and should be back online shortly</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>06:27</var> PST</small><br><strong>Investigating</strong> - We have been made aware of a core hardware router issue in one of our primary Datacentres in Vancouver, the team are working on a fix and we hope to have service restores shortly.</p>tag:servcomm1.statuspage.io,2005:Incident/200832542024-02-26T19:07:44-08:002024-02-26T19:07:44-08:003CX Service Affecting Issue - Partial Outage<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>19:07</var> PST</small><br><strong>Resolved</strong> - The incident has been resolved</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>16:33</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>16:32</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for further issues at this time...</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>16:06</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results, and will continue to monitor over the next few hours, please refresh your 3CX clients to ensure you reconnect if they have not automatically connected.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>16:00</var> PST</small><br><strong>Investigating</strong> - We have been made aware of a core hardware router issue in one of our primary Datacentres in Vancouver, the team are working on a fix and we hope to have service restores shortly.</p>tag:servcomm1.statuspage.io,2005:Incident/196716582024-01-10T16:43:59-08:002024-01-10T16:43:59-08:00Network Fault - Call completion and Audio Issues<p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>16:43</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>15:37</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>14:45</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>14:22</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented, and we are continuing to monitor services as they begin to recover fully</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>13:54</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>13:54</var> PST</small><br><strong>Update</strong> - Our Vendors have reported a fix is being implemented, call services should begin to return to normal shortly</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>13:38</var> PST</small><br><strong>Update</strong> - Further to our earlier update, we are now seeing reports of multiple Canadian carriers being affected, we will continue to work with our vendors to keep you apprised of the situation as it develops</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>13:33</var> PST</small><br><strong>Investigating</strong> - We are currently investigating a major fault with one of our third party vendors which is affecting call completion and progress, including poor call quality and inability to complete calls, we are working as quickly as possible for a resolution and appreciate your patience</p>tag:servcomm1.statuspage.io,2005:Incident/160001952023-02-07T08:23:18-08:002023-02-07T08:23:18-08:00SMS Registration Failing / Issues sending / receiving SMS MMS<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>08:23</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>16:09</var> PST</small><br><strong>Monitoring</strong> - We want to provide some additional details on the scope of this issue, and what you should expect.<br /><br />At the end of January, TCR released an update to their API that deprecated the auto-approval process for SMS Registration. Effective January 31, all new registrations require a manual review of the campaigns. Manual. As in humans are doing it. For every registration.<br /><br />Unfortunately, this change caused our registration system to break, as it needs to conform to these new requirements. In addition to changing our SMS Registration interconnections, customers will now have to wait while their Campaign Registration is manually reviewed before they can send SMS through our network. They have published an expected timeline of 5+ business days between the time a registration request is submitted and its approval.<br /><br />These changes have to be accounted for in how our portal interacts with the SMS Gateways, and is the reason for the extended resolution.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>13:48</var> PST</small><br><strong>Identified</strong> - We are aware of an issue that is preventing SMS Registration from completing successfully.<br /><br />We have determined the route cause to be related to a breaking API change made by The Campaign Registry. Unfortunately this change is not a small one, and will require some development work to accommodate. We do not have an ETA for resolution for this, but expect it to take at least a week to resolve.<br /><br />We are as frustrated by this as you are, and are working as fast as possible to resolve this issue.<br /><br />13:34 PST - 1 February 2023</p>tag:servcomm1.statuspage.io,2005:Incident/159435422023-01-26T11:40:16-08:002023-01-26T11:40:16-08:00Scheduled Maintenance - OVH Hosting<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>11:40</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>08:59</var> PST</small><br><strong>Update</strong> - Routine Maintenance now complete</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>08:35</var> PST</small><br><strong>Investigating</strong> - OVH Hosting services - Routine Maintenance<br /><br />Starts 09:00 PST<br />Services will be available as normal throughout</p>tag:servcomm1.statuspage.io,2005:Incident/159433402023-01-25T08:27:07-08:002023-01-26T08:11:08-08:00This is an example incident<p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>08:27</var> PST</small><br><strong>Resolved</strong> - Empathize with those affected and let them know everything is operating as normal.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>08:17</var> PST</small><br><strong>Monitoring</strong> - Let your users know once a fix is in place, and keep communication clear and precise.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>08:23</var> PST</small><br><strong>Identified</strong> - As you continue to work through the incident, update your customers frequently.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>08:53</var> PST</small><br><strong>Investigating</strong> - When your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.</p>