tag:status.video.ibm.com,2005:/historyIBM Video Streaming Status - Incident History2024-03-29T01:23:34-04:00IBM Video Streamingtag:status.video.ibm.com,2005:Incident/203128092024-03-21T11:06:59-04:002024-03-21T11:06:59-04:00VOD upload issue<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>11:06</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>17:02</var> EDT</small><br><strong>Monitoring</strong> - The video upload issue has been identified and a fix was implemented. We are now seeing videos that were stuck in "Queue" now being processed. Thank you for your patience. We will continue to monitor the issue.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>14:33</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>14:33</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating an issue to where video uploads remain in "Queued" Process". Video playback is not affected . <br />Thank you for your patience as we investigate.</p>tag:status.video.ibm.com,2005:Incident/201077452024-03-01T05:50:56-05:002024-03-01T05:50:56-05:00Emergency Maintenance on the World Wide Cluster<p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>05:50</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>04:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>09:59</var> EST</small><br><strong>Scheduled</strong> - During the maintenance window we’re going to restart services. Live and Vod playback could be affected during that period. We do not expect total downtime, but for some players it could cause blackscreen.</p>tag:status.video.ibm.com,2005:Incident/200913132024-02-29T10:16:02-05:002024-02-29T10:16:02-05:00US AWS Prod - Ingest 1.2 (No DB Migration / No Downtime) Cloud Ingest Upgrade<p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>10:16</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>09:59</var> EST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>09:54</var> EST</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>09:50</var> EST</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>09:37</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>16:17</var> EST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.video.ibm.com,2005:Incident/200622642024-02-26T13:35:48-05:002024-02-26T13:35:48-05:00Issues on our EU Cluster affecting Live Streaming and VOD playback<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>13:35</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>20:12</var> EST</small><br><strong>Investigating</strong> - Our engineering team is currently investigating an issue that's affecting Live Streaming and VOD playback on our EU Cluster. We will update this page as our engineers work on resolving the issue.</p>tag:status.video.ibm.com,2005:Incident/200320862024-02-22T10:55:02-05:002024-02-22T10:55:02-05:00US AWS Prod - VLM 29.16.2 (No DB Migration / No Downtime) Ingest Aspera and Kube DA<p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>10:55</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>10:49</var> EST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>09:46</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>14:55</var> EST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.video.ibm.com,2005:Incident/200386042024-02-22T04:00:56-05:002024-02-22T04:00:56-05:00Emergency Maintenance on the EU Cluster<p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>04:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>02:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>09:06</var> EST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.video.ibm.com,2005:Incident/199854802024-02-18T12:00:56-05:002024-02-18T12:00:56-05:00Emergency Maintenance on the World Wide Cluster<p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>12:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>08:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>12:39</var> EST</small><br><strong>Scheduled</strong> - During the maintenance window we’re going to restart services. Live and Vod playback could be affected during that period. We do not expect total downtime, but for some players it could cause blackscreen.</p>tag:status.video.ibm.com,2005:Incident/199870562024-02-15T11:40:46-05:002024-02-15T11:40:46-05:00US AWS Prod - VLM 29.16.1 (No DB Migration / No Downtime) Ingest<p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>11:40</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>16:41</var> EST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.video.ibm.com,2005:Incident/198394272024-02-14T11:45:59-05:002024-02-14T11:45:59-05:00Intermittent issues around webpage loading, streaming and chat<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>11:45</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>18:49</var> EST</small><br><strong>Monitoring</strong> - The issue is now resolved, and our engineering team is currently monitoring the situation.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>12:29</var> EST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>12:28</var> EST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>12:19</var> EST</small><br><strong>Investigating</strong> - We are currently investigating issues of intermittent issues around webpage loading , LIVE streaming and VOD viewing.</p>tag:status.video.ibm.com,2005:Incident/199128582024-02-12T05:00:57-05:002024-02-12T05:00:57-05:00Site load test<p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>05:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>03:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>09:30</var> EST</small><br><strong>Scheduled</strong> - We are going to preform a short load test on our video.ibm.com website.</p>tag:status.video.ibm.com,2005:Incident/198404672024-01-30T13:40:12-05:002024-01-30T13:40:12-05:00US AWS Prod - VLM 29.16.0 (No DB Migration / Downtime) EKS MXP Upgrade<p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>13:40</var> EST</small><br><strong>Completed</strong> - all testing completed</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>11:28</var> EST</small><br><strong>Verifying</strong> - still verifying release and testing</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>10:36</var> EST</small><br><strong>Update</strong> - system is down and release is in progress</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>09:32</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>14:40</var> EST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.video.ibm.com,2005:Incident/198191582024-01-25T21:03:43-05:002024-01-25T21:03:43-05:00US AWS Prod - VLM 29.15.2 (No DB Migration / No Downtime) DA EKS Upgrade<p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>21:03</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>21:03</var> EST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>10:52</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>10:50</var> EST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.video.ibm.com,2005:Incident/198240842024-01-25T07:49:37-05:002024-01-25T07:49:37-05:00Live Streaming issue on EU Cluster<p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>07:49</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>05:29</var> EST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>05:28</var> EST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>20:25</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>20:01</var> EST</small><br><strong>Investigating</strong> - We are reopening a past issue affecting our EU Cluster. Areas that are affected are:<br />-Live broadcast<br />-Live playlist<br />-Vod playback</p>tag:status.video.ibm.com,2005:Incident/198233452024-01-24T19:18:26-05:002024-01-24T19:18:26-05:00Live Streaming EU Cluster<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>19:18</var> EST</small><br><strong>Resolved</strong> - This incident has now been resolved.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>18:33</var> EST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>18:29</var> EST</small><br><strong>Investigating</strong> - We are currently investigating an issue affecting our EU Cluster for <br />-Live broadcast<br />-Live playlist<br />-Vod playback</p>tag:status.video.ibm.com,2005:Incident/196168462024-01-05T10:51:14-05:002024-01-05T10:51:14-05:00Network and technical issues<p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>10:51</var> EST</small><br><strong>Resolved</strong> - Our engineering team has confirmed that this issue is resolved and that the platform is back to full functionality. Please reach out to support should you experience issues. <br /><br />thank you for your patience as our team worked to fix and resolve this issue.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>00:58</var> EST</small><br><strong>Monitoring</strong> - Our engineers have applied a fix and we have regained site functionality. There is minor latency pulling up channel pages and connecting to the platform as our engineering teams continue to work on regaining full site functionality.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>20:30</var> EST</small><br><strong>Investigating</strong> - We are currently investigating the issue. Updates will be displayed as they become available.</p>tag:status.video.ibm.com,2005:Incident/194213112023-12-18T13:30:03-05:002023-12-18T13:30:03-05:00Akamai Configuration Migration Phase 2<p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>13:30</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>10:30</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>15:55</var> EST</small><br><strong>Scheduled</strong> - Our engineering team will be migrating the rest of our Akamai configurations during this time and it should not impact any broadcasting/streaming services.</p>tag:status.video.ibm.com,2005:Incident/194016942023-12-13T11:46:32-05:002023-12-13T11:46:32-05:00US AWS Prod - VLM 29.15.1 and Ingest 1.1 (No DB Migration / No Downtime) Transition DA and Ingest to Perpetual Aspera Lic<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>11:46</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>11:38</var> EST</small><br><strong>Update</strong> - Distributions are flowing Test Ingest files looked good</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>10:59</var> EST</small><br><strong>Verifying</strong> - Verifying DA and Ingest Aspera Lic chg</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>09:53</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>09:44</var> EST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.video.ibm.com,2005:Incident/194027742023-12-13T09:54:57-05:002023-12-13T09:54:57-05:00US AWS Prod - VLM New Ingest support upgrade<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>09:54</var> EST</small><br><strong>Completed</strong> - confirmed with nightly processing and things look good</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>06:43</var> EST</small><br><strong>Update</strong> - will continue to monitor till we see content drop</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>10:51</var> EST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>11:43</var> EST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.video.ibm.com,2005:Incident/193784752023-12-11T11:01:23-05:002023-12-11T11:01:23-05:00Akamai Configuration Migration<p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>11:01</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>09:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>12:05</var> EST</small><br><strong>Scheduled</strong> - Our engineering team will be migrating unused and/or low traffic Akamai configurations during this time and it should not impact any broadcasting/streaming services</p>tag:status.video.ibm.com,2005:Incident/192432972023-12-08T04:00:09-05:002023-12-08T04:00:09-05:00Maintenance window in legacy and eu-prod<p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>04:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>02:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>10:47</var> EST</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>11:25</var> EST</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance Dec 8th 2:00 -4:00 am EST. It is possible that LIVE and VOD playback may be affected in the player.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>11:20</var> EST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.video.ibm.com,2005:Incident/192450372023-12-07T18:26:08-05:002023-12-07T18:26:08-05:00US AWS Prod - VLM 29.15.0 (DB Migration / Downtime) DB Upgrade - Security fixes and DA chg<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>18:26</var> EST</small><br><strong>Completed</strong> - Partially successful needed to rollback Remote DA due to Java17 incompatible libraries</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>17:25</var> EST</small><br><strong>Update</strong> - still monitoring system</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>14:54</var> EST</small><br><strong>Update</strong> - reindexing tables to try and speed up the system - bringing system back down</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>12:56</var> EST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>09:32</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>15:55</var> EST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.video.ibm.com,2005:Incident/191202942023-11-17T07:00:26-05:002023-11-17T07:00:26-05:00Server Updates<p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>07:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>03:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>13:14</var> EST</small><br><strong>Scheduled</strong> - During the maintenance window the team will update and restart services both in legacy and eu-prod environments. Live playback and scheduling/stopping live playlists could be affected during the maintenance period. We do not expect total downtime, but for some players it could cause a black screen which can be resolved with a page reload.</p>tag:status.video.ibm.com,2005:Incident/173566452023-11-16T12:51:37-05:002023-11-16T12:51:37-05:00Intermittent Issues with Live Playlist.<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>12:51</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>25</var>, <var data-var='time'>11:43</var> EDT</small><br><strong>Monitoring</strong> - Our engineering team is currently monitoring our Live playlist and we will update this page with additional information asap</p><p><small>May <var data-var='date'>24</var>, <var data-var='time'>16:43</var> EDT</small><br><strong>Identified</strong> - We are currently monitoring an intermittent issue affecting our live playlist feature. Our engineers have noticed some anomalies and are currently investigating the root cause. We will update our status page as we hear more from our team. Thank you for your patience.</p>tag:status.video.ibm.com,2005:Incident/178483422023-11-16T08:58:56-05:002023-11-16T08:58:56-05:00VOD Indexing issue affecting Uploaded and Autorecorded videos<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>08:58</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>14:55</var> EDT</small><br><strong>Update</strong> - Our engineering team has identified and applied a fix to our storage and is continuing to restore access to affected videos. Should you have an urgent request for VOD prioritization, please reach out to IBM Video Streaming support.<br /><br />Videos that have already been restored will need to be re-published for viewing access.</p><p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>17:43</var> EDT</small><br><strong>Identified</strong> - There was an incident today that impacted the indexing of some videos in our storage, making those videos temporarily unavailable. If any of your videos were impacted, you will still see the videos listed in your dashboard, on channel pages or embedded players, but they will not be available for editing or playback. Our engineers are working on restoring access. Please subscribe to this issue to receive email updates.<br /><br />Thank you for your patience and understanding.</p>tag:status.video.ibm.com,2005:Incident/191100482023-11-15T03:15:29-05:002023-11-15T03:15:29-05:00Server Restarts<p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>03:15</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>03:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>03:00</var> EST</small><br><strong>Scheduled</strong> - During the maintenance window the team will restart services both in legacy and eu-prod environments. Live playback and scheduling/stopping live playlists could be affected during the maintenance period. We do not expect total downtime, but for some players it could cause a black screen which can be resolved with a page reload.</p>