Maintenance Bulletins

We need to make a change to our redis queue infrastructure in order to improve performance. This work should last from from 8:00:00AM to 09:00:00PM UTC and may cause delays or rejections of new screenshot requests intermittently. Existing cached requests will remain unaffected.

We will be performing a Kubernetes upgrade to address some issues that have been reported recently. We will also take this time to add updated Chrome browser instances to the pool. This work should last from from 10:00:00AM to 11:00:00AM UTC will most likely not cause any downtime.

We will be adding new GEO locations for our browsers to access which will involve restarting browser instances to add new location definitions. We will also be performing Kubernetes upgrade work to our master and worker nodes. This will allow us to use new Kubernetes features to improve scaling capabilities. This work should last from from 10:00:00AM to 11:00:00AM UTC and will cause intermittent downtime for new and cached requests.

We will be upgrading our browser infrastructure to allow for bigger containers to better process larger websites and full page images. This work should last from from 02:00:00PM to 03:00:00PM UTC and may cause intermittent rejection of new screenshots.

We will be performing database maintenance from 11:00:00PM to 12:00:00PM UTC. There may be queue delays while tables are locked during the maintenance process.

We are planning to build on the work we did splitting our monolithic redis instance into more single task redis instances to improve responsiveness. This task will begin at approx 13:00:00 UTC and finish at 14:00:00 UTC. There may be significant (tens of seconds) delays in processing screenshots and some screenshots may not process during this time as we use Redis to queue requests to our backend screenshot processors.

We are planning to split our Redis instances from one large Redis cluster instance to smaller instances for speed improvements and to take mitigate the single threaded nature of Redis. This task will begin at approx 07:00:00 UTC and finish at 08:00:00 UTC. There may be significant (tens of seconds) delays in processing screenshots and some screenshots may not process during this time as we use Redis to queue requests to our backend screenshot processors.

We're planning to move our remaining ScreenshotsCloud infrastructure over to Google Cloud Platform permanently. We've been testing this move by allowing gradually larger percentages of requests into GCP and will perform the final changeover on the 16th Feb. Some downtime and intermittent high screenshot processing time may be expected between 08:00:00 UTC and 09:00:00 UTC. This change will allow us to handle high demand while maintaining low processing times for screenshots.

We will be performing maintenance on the queue infrastructure which serves jobs to our browser instances starting 2017-12-14 08:00:00 UTC and ending no later than 2017-12-14 08:30:00 during which time there will likely be delays in serving browser rendering requests in a timely manner. This maintenance period is to allow us to provide regular maintenance to the queue system.

We will be performing maintenance on the queue infrastructure which serves jobs to our browser instances starting 2017-11-04 18:00:00 UTC and ending no later than 2017-11-04 18:30:00 during which time there will likely be delays in serving browser rendering requests in a timely manner. This maintenance period should provide a minor increase in rendering performance.

We will be performing maintenance on our load balancer starting 2017-10-28 02:00:00 UTC and ending no later than 2017-10-28 02:30:00. We expect no delays in serving browser rendering requests during this maintenance period.

We will be performing a major performance upgrade of our browser rendering infrastructure on 2017-10-01 12:00:00 UTC and ending no later than 2017-10-01 12:30:00. We expect minor delays in serving browser rendering requests during this maintenance period. Rendering times should be substancially reduced after this maintenance period.