Our data center location in Chicago, IL is experiencing some router issues - here is their news update:
We are currently experiencing some instability with one of our core routers in Chicago. In order to help relieve some CPU load off of this router, we have turned down some peering links in Chicago.
The current emergency maintenance should not result in any complete downtime, though as BGP convergence occurs, all customers may experience some localized downtime.
We are actively working to correct these issues and hope to have them resolved shortly.
Update (11:11AM 7/9/2010): We are rebooting this router in order to bring it back under control. Configuration changes have been made that should keep the issue under control after the reboot has completed.
Update (11:40AM 7/9/2010): Our core3.chi01 router has been rebooted and has returned to a normal state. core4.chi01 is currently working to recover after the core3 reboot and full network functionality should be restored shortly.
Update (12:12PM 7/9/2010): Our core4.chi01 router has failed to recover from the core3 reboot, and is currently experiencing heavy loads. We will be rebooting core4 to correct this issue.
We are working with Cisco to determine the root cause of this issue and plan to have a permanent fix in place soon.
Update (12:49PM 7/9/2010): core4.chi01 has been rebooted and is back online, but needs to be reconfigured, due to the device not properly retaining the configuration. This is in progress. We are estimating that full network connectivity should be restored within an hour.
Update (1:10PM 7/9/2010): core3 has been fully reconfigured. We are double checking the configuration to ensure that all customers and necessary services are properly online. We will update when everything is confirmed to be operational.
All times are Chicago time.
