The University of Chicago

The University of Chicago Status Announcement

Skip to: main navigation | utility navigation | main content

Back to the main announcement listing

IT Services  Green

04-15-2015 2:06 PM Green Please visit http://it-status.uchicago.edu/ for status updates

IT Services is no longer using this site to provide status updates. We have transitioned to a new vendor, StatusPage, which provides an offsite location for updates, as well as SMS and email subscription offers. To see the current status of services, please go to http://it-status.uchicago.edu.

For more information about subscribing to StatusPage alerts, see https://answers.uchicago.edu/page.php?id=46710.

03-18-2015 7:55 AM Green RESOLVED: Campus Connectivity Outage
Mar. 18, 2015 - At 6:46 a.m., internet connectivity was cut off between the campus and off-campus resources. Pre-existing connections usually remained open but new connections could not be opened across the campus border in either direction. ITS network engineers were able to isolate the problem to an error on the campus border firewall and restore normal service by 7:29 a.m.
03-04-2015 10:52 AM Green RESOLVED: Issue with Call Centers

Mar. 4, 2015 - As of 11 a.m. campus call centers are operating normally.

Recap of event: Between approximately 7:15 and 10:30 a.m. people trying to reach campus call centers may have experiencd long hold times or been unable to get through because of an issue on the back end call center servers. ITS engineers were able to re-synchronize the servers and normal service was restored gradually between 10:30 and 11 a.m.

03-04-2015 9:16 AM Yellow Issue with Campus Call Centers
Mar. 4, 2015 - We are currently experiencing an issue with campus call centers (including the ITS service desk) which may cause problems for people trying to reach those centers, including long hold times or an inability to get through. Email (or chat, if available) will still work normally. ITS engineers are working to resolve the issue.
02-18-2015 11:39 PM Green RESOLVED: Tank, Webshare, Business Objects, Website Disruption

Feb. 18, 2015 - As of 11:30 p.m., Tank, Webshare, Business Objects, and all websites are back up and running. Tank performance may be occasionally slow until disabled hardware can be fully repaired.

Recap of event: At approximately 3:15 p.m., a storage device failure disrupted a number of ITS services, including Tank fileshares; Webshare; and several campus websites (including news.uchicago.edu, cnet.uchicago.edu, and others). Additionally, Business Objects reporting service was disrupted due to dependency on Tank. ITS engineers and the vendor were able to restore service by 11:30 p.m.

02-12-2015 1:20 PM Green RESOLVED: Network Connectivity Issues

Feb. 12, 2015 - As of 1 p.m., campus network service is restored. Email to mailing lists was delayed during the outage and is currently being delivered.

Recap of event: At approximately 9:45 a.m., an equipment failure began causing network connectivity issues on the southeast side of campus. Buildings in that location had problems connecting to some services as well as general connectivity problems. Additionally, email to mailing lists was delayed, and VPN service was unavailable during the outage. ITS engineers are continuing to monitor the situation to make sure service remains stable.

01-27-2015 11:54 AM Green RESOLVED: Gargoyle/Student Systems Emergency Maintenance

Jan. 27, 2014 - As of 12 noon, Gargoyle, facultyaccess, staffaccess, Enrollment Manager, and classes.uchicago.edu are all now operating normally.

Recap of event: Overnight processing errors on the night of Jan. 26-27 caused data between Gargoyle and other applications to get out of sync. As a result, Gargoyle and related applications were brought down for emergency maintenance at 8 a.m. Jan. 27 to re-run the overnight processing. All applications were restored to normal service by 12 noon.

01-26-2015 3:37 PM Green RESOLVED - Workday Performance Issues

Jan. 26, 2015 - Workday is now operating normally.

Recap of event: On the morning of Jan. 26, Workday began experiencing intermittent performance issues due to a problem at the vendor's data center. The vendor was able to resolve the problem and restore normal service by 3 p.m.

01-16-2015 3:44 PM Green Workday Service Restored

Jan. 16, 2015 - Workday is now operating normally.

Recap of event: On the afternoon of Jan. 16, Workday began experiencing intermittent performance issues due to a problem at the vendor's data center. The vendor was able to resolve the problem and restore normal service by 3:35 p.m.

01-14-2015 6:49 PM Green RESOLVED: Network Latency Problem

Jan. 14, 2015 - As of 6:45 p.m., network service on campus is working normally.

Recap of event: At approximately 10:30 a.m. this morning, an issue arose on the campus network which caused slow performance or loss of connection for several applications, including Chalk, Delphi, Business Objects Reporting, cEikon, and some websites. People on the south side of campus also saw some generalized slow network performance, especially for off-campus connections. ITS network engineers were able to work around the problem and restore service by 6:45 p.m.

01-13-2015 10:01 PM Green RESOLVED: Campus Network Issue
Jan. 13, 2015 - At approximately 8:55 p.m. tonight, the campus network was cut off from the Internet. This includes connectivity to India, Beijing, and Hong Kong. Connectivity was restored at approximately 9:35 p.m. ITS engineers are monitoring the situation to make sure connectivity remains stable, and are investigating the source of the problem.
01-05-2015 10:49 AM Green Resolved: Login/Performance Issues with Workday

January 5, 2014 - As of approximately 10:15am the login/performance issue with Workday has stabilized and normal behavior restored.

Recap of event: Earlier this morning at approximately 8:30am Workday began experiencing an issue with the Portland, OR data-center where our Workday tenant is hosted. The issue with the data-center led to increased login times and/or a decrease in performance for many users. IT Services staff were in communication with Workday support about this issue and they addressed the issue at approximately 10:15am.

12-30-2014 12:46 AM Green Resolved: Auto Attendant Voicemail

Dec. 30, 2014 - As of 12:30 a.m. auto attendant voicemail is working normally.

Recap of event: At approximately 6 p.m. on Dec. 29th, a database for the legacy auto attendant voicemail application experienced a corruption of data. ITS engineers were able to restore the database from backup by 10 p.m., test all functionality and restore full service by 12:30 a.m.

11-14-2014 9:15 AM Green RESOLVED: cVPN issues

Nov. 14, 2014 - As of 9 a.m. cVPN is working normally.

Recap of event: Overnight on Nov. 13, cVPN stopped accepting connections from Mac users. Windows and other operating systems were unaffected. ITS engineers were able to repair a configuration problem and restore service by 9 a.m.

10-31-2014 11:06 AM Green UCTime Operating Normally

Oct. 31, 2014 - As of 11 a.m. UChicago Time (UCTime) is operating normally.

Recap of event: At approximately 10 a.m. on Oct. 31, the normal month-end load on UCTime caused slow performance on the application, sometimes to the point that some people could not log in. The vendor was able to correct the problem by 11 a.m.

10-24-2014 8:26 AM Green RESOLVED: Issues with cVPN Mac Connectivity

Oct. 24, 2014 - As of 8:15 a.m. cVPN is working normally.

Recap of event: At approximately 8 p.m. on Oct. 23, cVPN stopped accepting connections from Mac OS X clients. The issue was resolved at approximately 8:15 a.m. on Oct. 24 by failing over the cVPN system. ITS network engineers will continue to investigate the problem to ensure it does not recur. Windows and mobile clients were not affected.

09-01-2014 12:03 AM Green RESOLVED: Mainframe Maintenance Overrunning Window

August 31, 2014 - As of 11:55 p.m. the mainframe has returned to normal operations. Delayed jobs will be run overnight and regular overnight processing will occur.

Recap of event: The regular mainframe maintenance which was scheduled for this morning encountered unexpected problems and overran its window. Normal mainframe operation was restored at 11:55 p.m.

08-30-2014 10:24 AM Green RESOLVED: Incoming Calls to Campus Not Going Through

As of 10:30 a.m. phone service is working normally.

Recap of event: At approximately 3 a.m. this morning, a regional AT&T issue blocked incoming calls to campus phones and outgoing calls to toll-free numbers. Calls to the main Medical Center number were not affected. The issue was resolved by 10:10 a.m.

08-14-2014 10:09 PM Green RESOLVED: Network and data center power issue
Aug. 14, 2014 - As of 10:00 pm, all services and network connectivity have been restored. Chalk is online and functioning normally, but without full redundancy. IT Services is working to restore this backup system.
(Read More...)
08-05-2014 9:31 AM Green RESTORED - Degraded Voice, Network Service on North Quads

Aug. 5, 2014 - As of 9:15 a.m. voice and network service are working normally.

Recap of event: At approximately 12:15 a.m., a ComEd power issue at Eckhart Hall caused us to lose PBX voice and fax service on the North side of the main quads. We also lost all voice and network service in Eckhart. New VoIP phones outside of Eckhart were not affected. Facilities and ComEd were able to replace equipment and restore power by 9 a.m.; network and voice service were fully restored by 9:15 a.m.

06-19-2014 8:22 AM Green RESOLVED: answers.uchicago.edu knowledgebase unavailable

June 19, 2014 - As of 1:00 a.m. on June 19, 2014 the answers.uchicago.edu knowledgebase has been restored to normal operation.

Recap of event: At approximately 8:20 a.m. on June 18, 2014, the answers.uchicago.edu knowledgebase became unavailable due to a network problem at the off-campus provider which hosts the service for us. ITS engaged with the provider and answers.uchicago.edu was partially restored by 10:30 a.m. on June 18, 2014, but content was read-only while full recovery was in progress. As of 1:00 a.m. on June 19, 2014 full service was restored and content was once again able to be edited.

05-16-2014 5:55 PM Green RESOLVED: Network issues affecting campus

May 16, 2014 - As of 5:45pm all services have been restored.

Recap of event: At approximately 2:45pm this afternoon we experienced a networking outage at one of our data centers. Some of the services which may have been affected include email services, HR systems (including payroll), authentication services, secure wireless, instant messaging, Chalk, my.uchicago.edu and web sites hosted at the university. Most services were restored shortly before 4:00pm, with wireless restored at 4:30pm and VPN restored at 5:00pm. Employee self service (ESS), delphi budget reporting and business objects reporting were restored at 5:45pm.

05-14-2014 3:14 PM Green RESOLVED: Issues with Secure Wireless Service

May 14, 2014 - As of 2:45 p.m. the secure wireless networks - uchicago-secure and eduroam are once again fully operational.

Recap of event: At approximately 1:40 pm this afternoon ITS began receiving reports of problems connecting to the secure wireless networks - uchicago-secure and eduroam across campus.  ITS Network Engineers discovered a component of the secure wireless network was not properly handling authentication, preventing new connections to either network. Individuals who were already connected were not affected. ITS was able to restore service by switching to a backup server, allowing authentication to resume. ITS Network Engineers will be working with the vendor to identify a fix and prevent this from recurring.

05-12-2014 12:01 PM Green RESOLVED: Hinds Data Center Power Outage

May 12, 2014 - As of 11:45 a.m., the Hinds Data Center is back on-line and fully operational.

Recap of event: At approximately 3 a.m. this morning, a power issue in the Hinds Data Center led to a failure with network equipment in the area, resulting in wired and wireless network being down in Hinds and the Center for Integrative Science. Wired and wireless connectivity was restored shortly before 8 a.m., however the Hinds Data Center remained off-line while full power, lighting and cooling was restored. All services were fully restored by 11:45 a.m.

04-29-2014 12:33 PM Green RESOLVED: Wireless Outage
Apr. 29, 2014 - At approximately 12 noon, a wireless controller on campus spontaneously rebooted due to a software bug. Wireless access on campus was disrupted for approximately 15 minutes while the service failed over. ITS engineers will be working with the vendor to identify a fix and prevent this from recurring.
04-14-2014 12:59 PM Green RESOLVED: Widespread Systems Outage

Apr. 14, 2014 - As of 1 p.m. email is back up, though some users may still see slow performance for a while as their mailboxes finish resynchronizing. All other services are operating normally.

Recap of event: At approximately 4 a.m. a power equipment failure at the 6045 data center caused a widespread service outage, including email, secure wireless, web authentication, my.Uchicago, and several administrative applications. Service was fully restored by 1 p.m.

04-03-2014 3:05 PM Green RESOLVED: Network Issues on Campus

Apr. 3, 2014 - As of 1:30 p.m. network service on campus is operating normally.

Recap of event: At approximately 6 a.m. this morning, a power spike related to the concurrent electrical storm led to a failure of network equipment in several buildings on campus, including Burton Judon Courts, Erman Biology, Rockefeller Chapel, Judd Hall, and 5736 Woodlawn. By 10 a.m. service was restored to all sites except Erman and Rockefeller; service to those two buildings was restored by 1:30 p.m.

03-21-2014 8:05 PM Green RESTORED: Voicemail Service

Mar. 21, 2014 - As of 7:45 p.m. voicemail is functioning normally.

Recap of event: At approximately 4:30 p.m., a power issue caused a legacy voicemail system still used by several departments on campus to crash. ITS voice engineers were able to work with the vendor to fully restore service by 7:45 p.m.

02-11-2014 3:22 PM Green RESOLVED: Issues with DHCP (wireless and wired network)

Feb. 11, 2014 - As of 3:15 p.m. all network services are operating normally.

Recap of event: At approximately 2 p.m. a CPU spike on our primary DHCP server caused it to fail. Since DHCP provides dynamic network addressing to wireless devices and many wired devices on campus, the dependent devices began to lose their network connections beginning around 2:15 p.m. ITS network engineers were able to force a fail over and restore full service by 3:15 p.m., and will be investigating the cause of the CPU spike to ensure this does not happen again.

11-27-2013 10:38 AM Green RESOLVED: UChicagoTime time recording application inaccessible

November 27, 2013. Access to UchicagoTime has been restored by the vendor.

Recap of event: Routine overnight maintenance by the vendor of UChicagoTime exceeded the scheduled window, leaving the application inaccessible. The service was restored for staff submitting their time at approximately 10:00am with full restoration around 12:00pm.

11-14-2013 2:41 PM Green Resolved: Problems with International Calls from Campus Phones

Nov. 14, 2013 - As of 1:45 p.m. international calling is working normally.

Recap of event: At approximately 12 noon, an issue at the University's long-distance provider began causing international calls placed from campus phones to fail. The provider was able to restore service by 1:45 p.m. Domestic long-distance and local calls were unaffected.

10-09-2013 12:20 PM Green RESOLVED: Problems accessing off-campus sites on eduroam and uchicago-secure

Oct. 9, 2013 - As of 12 noon, wireless service is working normally.

Recap of event: At approximately 10 a.m., some users on the uchicago-secure and eduroam wireless SSIDs stopped being able to access off-campus sites. This was a cascade effect of resolving general network issues earlier this morning. ITS network engineers will be investigating the entire chain of events to develop processes that will prevent this from happening again.

10-09-2013 10:43 AM Green RESOLVED: Network Connectivity Issues

Oct. 9, 2013 - As of 10:30 a.m. all services apart from some wireless access have been restored.

Recap of event: At approximately 7:45 a.m. a network hardware issue caused some users to have trouble accessing network services on the south side of campus, as well as VPN connections from off-campus. Cascade effects from resolving this problem caused caused campuswide disruption for some specific applications. As of 10:30 a.m. all application access has been restored, though a separate issue still exists for uchicago-secure wireless.

07-16-2013 11:00 AM Green RESOLVED: sporadic issues with uchicago-secure, eduroam
Jul 16, 2013 - Beginning at approximately 10am, IT Services received reports that some but not all users of secure wireless (uchicago-secure and eduroam) were unable to access off-campus sites. Network engineers restored functionality by 10:45am and continue to investigate the underlying cause.
06-20-2013 7:39 AM Green RESOLVED: Intermittent issues with local area code calls
Jun. 20, 2013 - As of midnight, AT&T has resolved the issues in their infrastructure and phone service is functioning normally. Recap of event: Beginning at approximately 1pm Jun 19 2013, calls over AT&T in Chicago randomly failed or experienced poor call quality. AT&T identified the issue in their infrastructure which required several hours to fully repair. IT Services has made several test calls to and from the university and hospitals to confirm the repair.
06-17-2013 10:12 AM Green Authentication Services Restored
Jun. 17, 2013 - At approximately 9:40 a.m. a cooling failure in one of our data centers caused a network device to fail. This caused a disruption for LDAP-related authentication, which prevented people from logging in to wireless, wiki, mailing lists, and other services. The device was successfully failed over and service restored by 9:50 a.m. IT Services technicians will be investigating the root cause of the incident to prevent it from happening again.
05-29-2013 5:57 PM Green UPDATE - Inbound calls to some Gleacher phones fail

May 29, 2013 - As of 5:30 p.m., critical phone numbers at Gleacher have been re-routed to a central number. Outbound calls continue to work normally. IT Services is working with the carrier for our circuit to resolve the issue as soon as possible.

Recap of event: At approximately 6:45 a.m., phone numbers (312)423-xxxx at Gleacher stopped receiving inbound calls. Outbound calls and all other phone numbers are working normally. As of 1:30 p.m. critical phone numbers at Gleacher were re-routed to a central number to allow partial restoration of incoming service, while IT Services continued to work with the carrier to fully restore service.

05-12-2013 7:19 AM Green Scheduled Database Maintenance: Complete
May 12, 2013 - The upgrade to our primary administrative database server is complete.
05-08-2013 1:20 PM Green Wireless Service Interruption
May 8, 2013 - At approximately 12:40 p.m. a hard drive failure on a wireless gateway device disrupted non-secure wireless (SSID: uchicago) on the North side of campus. ITS technicians were able to manually fail over to a redundant device and restore service by 12:55 p.m. Secure wireless (SSID: uchicago-secure), Eduroam, and the Guest network were unaffected.
04-30-2013 2:26 PM Green Intercall Service Restored

Apr. 30, 2013 - As of 2:15 p.m., access to the Intercall audioconference service has been fully restored. If you are still seeing issues, please contact the ITS Service Desk at 2-5800.

Recap of event: At approximately 1 p.m., the toll-free dial-up number to the University's audioconference vendor, Intercall, stopped working. Callers were unable to connect consistently via the toll-free number. As of 2:15 p.m. Intercall believes that the issue has been resolved; ITS's tests are validating this belief.

04-22-2013 9:03 PM Green CTV Maintenance Complete
Apr. 22, 2013 - As of 9 p.m. the emergency maintenance on CTV is complete. CTV service to both wired internet connections and set-top boxes is fully restored.
04-19-2013 6:17 PM Green cTV issue ongoing; other services operating normally
Apr. 19, 2013 - As of 6 p.m., service to most cTV set-top boxes has been restored. IT Services will continue to work over the weekend to fully restore set-top box service. In the meantime, cTV continues to be available via a web browser from any computer using the wired network.
03-26-2013 8:14 AM Green RESOLVED: Several services in IT Services data center disrupted
March 26, 2013 - Beginning at 7:20am, high network traffic resulted in severe degradation or unavailability of several services in the IT Services data centers. The source of the traffic volume was isolated, restoring normal service by 7:50am.
03-08-2013 11:36 AM Green RESOLVED: Phone Issues at Gleacher and 401 N Michigan
Mar. 8, 2013 - As of 11 a.m., phone service to and from Gleacher and 401 N Michigan is operating normally. Recap of event: At 7:50 a.m., IT Services began receiving reports that clients at the Gleacher location of the Booth School of Business were unable to place outbound phone calls, and that some inbound calls were also failing. Later reports confirmed that 401 N Michigan was experiencing the same problem. ITS technicians traced the problem to a configuration error in a network switch and fully restored service by 11 a.m.
02-10-2013 12:39 PM Green Power Restored
Feb. 10, 2013 - As of 12:30 p.m., power has been restored to campus and the network in the affected locations has been fully restored. Recap of event: At approximately 10:30 a.m., the south side of the Quads and some buildings east of the Quads between 58th and 59th St. lost electrical power; power was restored at approximately 12:30 p.m. No ITS services were affected outside the buildings which lost power.
01-28-2013 10:51 AM Green RESOLVED: Voice Conferencing Outage

Jan. 28, 2013 - As of 10:45 a.m., the University's primary voice conferencing service is fully restored and operating normally.

Recap of event: At approximately 8:15 a.m., ITS became aware that the University's primary voice conferencing service was unavailable. The problem was tracked to a vendor error; the vendor was able to reactivate our conference codes and fully restore service by 10:45 a.m.

01-10-2013 2:52 PM Green RESOLVED: Data Center Connectivity

Jan. 10, 2013 - As of 2:30 p.m. all services are fully restored and operating normally.

Recap of event: At approximately 12:45 p.m. a network hardware problem in one of the data centers began causing sporadic or inconsistent connection problems to a number of services. By 1:45 p.m. xMail and Gargoyle, among other services, had become completely unavailable; people also began having problems authenticating to secure wireless, as well as problems with some telephones or voice features. ITS technicians were able to fully restore service by 2:30 p.m.

01-09-2013 9:45 AM Green RESTORED: xMail and Sharepoint
Jan. 9, 2013 - As of 9:30 a.m., Sharepoint and xMail are operating normally. Recap of event: At approximately 6:45 a.m., a Microsoft patch applied this morning to xMail and Sharepoint began causing performance problems which were not seen in pre-install testing. This included problems with web-based connections, issues with Mac OS X email clients, and error messages related to autodiscovery in other clients. IT Services technicians were able to roll back the patch and restore service by 9:30 a.m.
01-09-2013 6:59 AM Green RESTORED: Network Performance Issues

Jan. 9, 2013 - As of 4:30 a.m., network service has been fully restored.

Recap of event: At approximately 2:45 p.m. on Jan. 8, the University lost connectivity to our two primary research networks and two of our three Internet service providers due to a major communications cable getting cut in downtown Chicago, which also affected Argonne and several other universities in Illinois. As a result people may have seen a brief disruption as connections failed over, followed by slow performance for off-campus connections because of reduced bandwidth. The provider was able to repair the cable and restore full service by 4:30 a.m. Jan. 9.

12-14-2012 1:11 AM Green RESOLVED: Tank Fileshares

Dec. 14, 2012 - As of 1 a.m., Tank file share access is fully restored, and Tank is operating normally.

Recap of event: On the afternoon of Dec. 13, IT Services became aware of a significant problem on the Tank file share service. At approximately 2:45 p.m., access to Tank was blocked so ITS technicians could investigate the situation and how best to mitigate it. The problem was resolved, and Tank access fully restored, by 1 a.m. Dec. 14.

12-10-2012 11:00 PM Green RESOLVED: Network Name Server Issues
Dec. 10, 2012 - At approximately 9:45 p.m. this evening CPU load on the network name servers came under an unusually high load; as of approximately 10:15 p.m. the load reached a point that made the service unavailable. As a result people began having trouble reaching web, email, and other services on the Internet. ITS technicians were able to restart the servers and restore service by approximately 10:45 p.m. ITS will continue to monitor the service closely and will be consulting the vendor to track down the source of the problem and ensure it does not happen again.
12-07-2012 9:24 AM Green RESOLVED: Slow Web/Email Performance

Dec. 7, 2012 - As of 9 a.m., the vendor issues which were causing slow performance on some SSL-using services (web and email) seem to be fully resolved.

Recap of event: Early on the morning of Dec. 6, a problem at one of the major commercial providers of Internet trust services (e.g. SSL security certificates) experienced a problem which interfered with their ability to authenticate secure sessions. The vendor resolved the issue within two hours, but Internet service providers, individual services, and individual user machines using cached information continued to see slow performance until they had locally cleared bad cached information, a process which took several hours.

Detailed information on the actual outage and sequence of events is scant, but interested readers might find this brief article useful:

https://isc.sans.edu/diary/Comodo+DNS+hiccup+on+usertrust+com/14653

11-09-2012 12:05 PM Green RESTORED: Intermittent Voicemail Issues
November 9, 2012. As of noon, IT Services technicians have stabilized the voice mail system. They continue working with the vendor to determine the root cause. Recap of event: Shortly after 6:00 this morning, voicemail servers began experiencing service failures, causing intermittent failures of the voice mail service on campus and in the medical center. The system was restored to stability by noon.
11-02-2012 6:13 PM Green RESTORED: Service Outage

Nov. 2, 2012 - As of 6 p.m., all services are operating normally. Outlook users may need to restart their email client to fully re-connect to the mail server.

Recap of event: At approximately 4:30 p.m. a network configuration error caused a number of servers in the primary ITS data center to be cut off from the network. Most services failed over to redundant servers but some Outlook connections were briefly disrupted and a number of web servers became unavailable. ITS staff were able to restore the configuration and fully restore all services by 6 p.m.

11-02-2012 9:19 AM Green RESTORED: UChicago Time
Nov. 2, 2012 - As of 6:00 am this morning, the UChicago Time and Attendance System has returned to normal operations. If you have any questions or concerns, please contact the UChicago Time support team at uchicagotimesupport@uchicago.edu.
10-28-2012 6:30 AM Green COMPLETED: Wireless Network Maintenance
October 28, 2012 - At 6:25 a.m., the maintenance on the wireless controllers was completed across the Hyde Park, Gleacher Center, and 401 N. Michigan campuses and wireless service has been restored to full operation.
09-23-2012 8:53 AM Green cVPN Restored
Sep. 23, 2012 - As of 8:45 a.m. cVPN is operating normally. Recap of event: At approximately 3:30 a.m. a software issue caused cVPN to become unavailable. IT Services technicians were able to repair the configuration and restore service by 8:45 a.m., and will continue to work with the vendor to determine the root cause of the issue.
09-22-2012 10:38 AM Green RESTORED: cVPN

Sep. 22, 2012 - As of 10:30 a.m. cVPN is operating normally.

Recap of event: At approximately 7 a.m., a software corruption issue caused cVPN to stop accepting logins. ITS technicians were able to restore the configuration and restore service by 10:30 a.m.

09-16-2012 9:49 AM Green RESTORED: Web Single Sign-On (myUchicago)

Sep. 16, 2012 - As of 9:30 a.m. web single sign-on (used by myUchicago, gmail, Chalk, and other web-based services) is working normally.

Recap of event: At approximately 8:15 a.m., during a normal network upgrade, web single sign-on stopped processing logins. IT Services technicians were able to restart the service and restore service by 9:30 a.m.

09-14-2012 9:29 AM Green RESOLVED: Non-Secure Wireless Outage

Sep. 14, 2012 - As of 9:15 a.m. wireless networking on campus is operating normally.

Recap of event: At approximately 8 a.m., the authentication servers for non-secure wireless (SSID "uchicago") stopped processing logins. IT Services technicians were able to restart the servers and restore access to the main campus by 9 a.m.; service was fully restored by 9:15 a.m. ITS will continue to investigate why this event happened to prevent its recurrence.

09-14-2012 8:40 AM Yellow Non-Secure Wireless Outage
Sep. 14, 2012 - IT Services has become aware that people are unable to connect to the non-secure wireless network (SSID "uchicago"). IT Services staff are working to resolve the problem. Secure wireless options (SSID "uchicago-secure" and "eduroam") are working normally campuswide; see http://answers.uchicago.edu/search.php?q=&cat=760 for information on migrating to that option.
08-30-2012 10:06 AM Green RESTORED: Xmail availability
Aug. 30, 2012 - Xmail is fully functional for all users. At approximately 6:45 a.m. today, a database issue caused about 75 users to temporarily lose access to xmail. No mail was lost while IT Services technicians restored the database. Service was restored by 9:50 a.m.
08-15-2012 4:52 PM Green RESTORED - Law School Network
Aug. 15, 2012 - As of 4:45 p.m., the network is operating normally. Recap of event: At approximately 3:15 p.m., a switch configuration problem caused the Law School to lose telephone and data network service (both wired and wireless). IT Services technicians were able to repair the configuration and restore service by 4:45 p.m.
08-10-2012 9:41 PM Green RESTORED: xMail, web authentication, other services
Aug. 10, 2012 - Between 8:45 and 9 p.m., a network connection issue caused a number of services to become unavailable. Affected services included xMail and web single sign-on (to myUchicago, Library databases, and other resources). IT Services technicians were able to restore connectivity and as of 9 p.m. all services are operating normally.
08-02-2012 3:46 PM Green RESTORED: Lists, Bulkmail

Aug. 2, 2012 - As of 3:30 p.m., lists.uchicago.edu and bulkmail.uchicago.edu are working normally. There was a brief service disruption for Chalk during the restore process, but it recovered within five minutes and is also operating normally.

Recap of event: At approximately 12:45 p.m. today, a storage problem caused lists.uchicago.edu and bulkmail.uchicago.edu to become unavailable. IT Services technicians were able to migrate to new storage and recover service by 3:30 p.m.

08-02-2012 8:48 AM Green RESOLVED: Secure Wireless authentication failure
RESOLVED: Secure Wireless authentication failure Aug. 2, 2012 - Early this morning, IT Services became aware that people had been unable to authenticate to secure wireless (SSID "uchicago-secure" and "eduroam") since late evening on August 1. IT Services technicians re-established service on a backup server by 8:00 a.m. and continue to investigate the failure on the original server. Users may be asked to accept a certificate from the server netauth-01 to connect to secure wireless.
08-01-2012 12:47 PM Green RESOLVED: Non-Secure Wireless on South Campus

Aug. 1, 2012 - As of 12:30 p.m., the campus wireless network is working normally.

Recap of event: At approximately 8:15 a.m. IT Services began receiving reports that people were unable to connect to non-secure wireless (SSID "uchicago") on the south side of campus (south of 58th St). The vendor traced the issue to a problem with the non-secure wireless authentication process, and was able to restore service fully by 12:30 p.m. IT Services will continue to work with the vendor to determine why this happened and prevent its happening again.

07-27-2012 9:48 AM Green RESOLVED: Intermittent Voicemail Issues
July 27, 2012 - At approximately 8:10 a.m. one of the four nodes used to distribute load on the voicemail system experienced a failure. ITS technicians were able to restore normal service by 9 a.m. During the time of the outage, users attempting to leave or access voicemail may have been unable to connect on their first attempt. ITS is working with the vendor to identify the exact cause of the failure and remedy it.
07-23-2012 3:42 PM Green Online directory briefly unavailable
RESOLVED: Online directory service disruption Jul. 23, 2012 - At approximately 3:10 p.m. this afternoon, a server failure caused the online directory service to be unavailable for approximately 20 minutes IT Services staff are working to prevent this from happening again.
07-11-2012 4:49 PM Green RESOLVED: answers.uchicago.edu Unavailable

July 11, 2012 - As of 4:30 p.m., the IT Services Knowledge Base at https://answers.uchicago.edu is working normally.

Recap of event: At approximately 10:15 a.m., answers.uchicago.edu became unavailable due to a network storage problem at the off-campus provider which hosts the service for us. The provider resolved the issue and restored full service by 4:30 p.m. ITS will be engaging the provider in discussions about mirroring the site in other locations to help prevent this situation from happening again.

07-11-2012 10:51 AM Green RESOLVED: Off-Campus Network Issues

July 11, 2012 - As of 10:45 a.m., off-campus network connectivity is fully restored.

Recap of event: At approximately 8:55 a.m., a hardware failure in one of the core network switches caused us to lose one of our off-campus network connections. This resulted in a 2-3 minute outage while traffic was automatically re-routed; additionally, we were at reduced bandwidth for off-campus connections until the hardware was replaced. ITS technicians replaced the faulty hardware and were able to fully restore the off-campus network connectivity by 10:45 a.m.

07-10-2012 12:34 PM Green RESOLVED: xMail Login Issue

July 10, 2012 - As of 12:30 p.m., xMail logins are working normally.

Recap of event: Early in the morning of July 10, one of the login servers for the xMail service began experiencing an exceptionally high load. As a result, many Outlook users began having trouble logging in. Service was mostly restored by 11 a.m. by removing the problem server, and fully restored by 12:30 p.m. ITS technicians continue to work to isolate the cause of the extra load and ensure it does not recur.

07-06-2012 1:49 PM Green RESOLVED: Call Center Problems
July 6, 2012 - As of approximately 1:20 p.m., all campus telephone service is operating normally. Recap of event: At approximately 1 p.m. this afternoon the primary call center system for campus malfunctioned, causing several call centers on campus and in the medical center to stop working (callers would get a busy signal or a disconnect instead of hearing an auto-attendant or voicemail tree option). ITS technicians were able to fail over to a backup server by 1:20 p.m. and continue to work with the vendor to ensure that the system is stable.
06-21-2012 1:56 PM Green RESOLVED: xMail, Gargoyle, other service disruption
Jun. 21, 2012 - At approximately 1:06 p.m. this afternoon, a hardware failure in the 6045 data center caused several services hosted there to be cut off from the network; affected services included xMail, Gargoyle, PowerFAIDS, and some telephony call management services, among others. Systems recovered at approximately 1:12 and all services were available again by 1:20. IT Services staff are working to prevent this from happening again.
06-20-2012 11:37 AM Green RESOLVED: Login issues on non-secure wireless
Jun. 20, 2012 - As of 11:30 a.m., the wireless network is accessible and working normally. Recap of event: At approximately 10:10 a.m., the login page for non-secure wireless (SSID: uchicago) on the south side of campus (south of 58th St) stopped working. ITS technicians traced the issue to a configuration error on the wireless gateway and were able to restore normal service by 11:30 a.m.
06-16-2012 4:29 AM Green RESOLVED: Exchange Disruption
June 16, 2012 - At approximately 2:30 a.m., a power issue caused some of the back-end servers for xMail to fail. xMail was available for most users again by 3 a.m., and was fully recovered by 4:15 a.m.
06-15-2012 9:38 PM Green RESOLVED: Gargoyle, xMail Disruption
June 15, 2012 - As of 9:30 p.m. Gargoyle and xMail are working normally. Recap of event: At approximately 8 this evening, a mechanical failure caused loss of cooling in the IT Services data center at 6045 S. Kenwood; this caused a number of services to shut down to prevent data loss, most notably Gargoyle and xMail. Cooling was restored by 9 p.m. and all services were restored by 9:30.
06-10-2012 8:26 AM Green RESOLVED: network connectivity from off-campus
June 10, 2012. After a brief disruption this morning, the campus network is communicating normally with off-campus locations. The outage began at approximately 7:50am 6/10/12 and ended at approximately 8:15am. In that interval, the campus network was isolated from any traffic from outside of campus. Engineers are investigating the cause of the issue.
06-06-2012 5:30 PM Green RESOLVED: Non-secure wireless outage on North Campus

June 6, 2012 - As of 5:25 p.m., wireless on campus is operating normally.

Recap of event: At approximately 4:45 p.m., one of the wireless gateways experienced a hard crash, causing non-secure wireless (SSID: uchicago) to stop working on the north side of campus (north of 58th St). IT Services technicians manually restarted the gateway and service was restored by 5:20 p.m.

05-31-2012 11:58 AM Green RESOLVED: Telephone Service Issues
May 31, 2012 - As of 11:45 a.m. telephone service is working normally. Recap of event: On the morning of May 31, a hardware issue in part of the central telephone switching system caused some campus phones to experience loud static on the line for different functions (incoming and/or outgoing calls, transferring calls, putting calls on hold). ITS staff were able to partially mitigate the problem with redundant hardware at approximately 11 a.m., and the incident was fully resolved by 11:45 a.m.
05-23-2012 1:35 AM Green RESOLVED: my.UChicago, Library databases, other services
May 23, 2012 - As of 1:30 a.m., all services should be operating normally. Recap of event: At approximately 10:45 p.m. on May 22, a storage problem on one of the University's central authentication servers caused all applications dependent on that authentication server to become unavailable; affected services included the my.UChicago portal, the Library proxy and several Library databases, and the AURA Grants system, among others. IT Services engineers were able to resolve the storage issue by 1:30 a.m. May 23.
05-14-2012 6:11 PM Green RESOLVED: Power Loss on South Quads Affecting Phone Service

May 14, 2012 - As of 6 p.m., ComEd has restored power and all network and telephone service is restored.

Recap of event: At approximately 4:45 p.m., several buildings on campus lost power. Since one of the affected buildings also houses central telephone switches for the South side of the main Quads, buildings without power and other buildings on the South side of the Quads lost phone service when battery backup ran out between 5 and 5:15 p.m. ComEd restored power by approximately 5:40 p.m., and by 6 p.m., all network and phone service was operating normally.

05-09-2012 9:01 AM Green RESOLVED: Power failure causes network outage for parts of campus
May 9, 2012 - As of 8:55am all network gear is again functioning normally after a power hit at 7:40am May 9 2012 caused a loss of connectivity to some areas of campus, including Crerar Library and parts of the hospital.
04-25-2012 9:20 AM Green RESOLVED: Outlook Connections from Off-Campus
Apr. 25, 2012 - As of 5 a.m., Outlook is working normally from off campus. Recap of event: An upgrade to a network interface caused some Outlook configurations to fail when connecting to xMail from off-campus. On the morning of Apr. 25, the upgrade was rolled back; Outlook should now be working normally. If you are still having trouble, contact the ITS Service Desk at 773-702-5800, via email at itservices@uchicago.edu, or get walk-in help at the TECHB@R on the first floor of Regenstein Library during reference desk hours.
04-20-2012 8:32 PM Green RESOLVED: xMail/Gargoyle Disruption
Apr. 20, 2012 - As of 8:20 p.m., xMail and Gargoyle are operating normally and with full network redundancy. Recap of event: At approximately 2 p.m., a network interface malfunctioned in a way which caused xMail and Gargoyle to begin sporadically losing their network connections. Gargoyle resolved itself by approximately 2:15 p.m., but xMail continued to have issues until the problem device was disconnected at approximately 2:45 p.m., restoring xMail service but leaving us without network redundancy. ITS technicians continued to work with the vendor to track down the configuration issue which caused the outage and restored full network redundancy at approximately 8:20 p.m.
04-18-2012 8:21 AM Green RESOLVED: xMail Web Interface Outage
Apr. 18, 2012 - As of 8:10 a.m., the xMail web interface is operating normally. Recap of event: An upgrade to a network interface this morning led to some people being unable to log in to the xMail email service over the web. The issue was tracked to a configuration issue and resolved by 8:10 a.m. If you attempted to connect during the outage, you may need to clear your browser cache before reconnecting; if you are unsure how to do this, contact the ITS Service Desk at 773-702-5800 or itservices@uchicago.edu.
04-15-2012 6:48 AM Green COMPLETED: Wireless Network Maintenance
April 15, 2012 - At 6:40 a.m., the maintenance on the wireless controllers has been completed across campus and wireless service has been restored to full operation.
04-10-2012 3:09 PM Green RESOLVED: cMail Performance
Apr. 10, 2012 - As of 3 p.m., cMail is working normally. Recap of event: At approximately 2 p.m. Tuesday, Apr. 10, a sudden spike in incoming email overloaded the back-end cMail mail servers. As a result, cMail users began experiencing poor performance, or were unable to connect. Between 2:30 and 2:45 IT Services technicians restarted the back-end service, returning performance to normal.
03-09-2012 10:11 PM Green RESOLVED - xMail, other service disruption
Mar. 9, 2012 - As of 9:45 p.m. all services are operating normally. Recap of event: At approximately 8:20 p.m., a network issue caused several services in the 6045 Data Center (including xMail and Gargoyle, among other services) to began having severe performance issues and become unavailable. ITS staff were able to restart problem devices and restore service by 9:45 p.m.
03-09-2012 4:36 PM Green RESTORED: cTV
Mar. 9, 2012 - As of 4:15 p.m., cTV is operating normally for web clients and most set-top boxes. A few set-top boxes are still seeing issues, but this appears to be due to a different problem; IT Services staff are working with the vendor to identify and resolve the issue for those boxes.
02-15-2012 4:26 PM Green RESTORED: my.Uchicago
Feb. 15, 2012 - As of 4:15 p.m., my.UChicago, the University's central web portal, is available and operating normally. Recap of event: At approximately 1:45 p.m. on Feb. 15, a file storage problem on my.UChicago caused the system to become unavailable. IT Services technicians were able to restore the files and restore service by 4:15 p.m.
02-14-2012 1:48 PM Green RESOLVED: Problems with Searches on UChicago Websites
Feb. 14, 2012 - As of 1:30 p.m., all UChicago website searches should be operating normally. Recap of event: At approximately 9:30 a.m., the Google Search Appliance supplying search capabilities for many University websites experienced a software failure. The Appliance failed over to a backup server, but as the backup server did not have completely current data, searches on University websites other than www.uchicago.edu did not always return the expected results. ITS technicians re-synchronized the backup server with current data and all website searches were operating normally by 12 noon.
02-13-2012 3:48 PM Green RESTORED - Wireless on North Campus
Feb. 13, 2012 - As of 3:45 p.m., wireless on North Campus appears to be stable and operating normally. Recap of event: At approximately 3 p.m., a wireless authentication server for the north side of campus (north of 58th St) experienced a failure. As a result, users on this part of campus became unable to log in to the "uchicago" wireless network. The "uchicago-secure" and "eduroam" wireless networks were not affected. The server was restarted and service restored by 3:45 p.m. ITS engineers will be working with the vendor to trace the cause of the failure and put in a permanent fix.
02-13-2012 9:28 AM Green RESTORED - Blackberry Service
Feb. 13, 2012 - At approximately 5:30 a.m., the Blackberry server attached to xMail, the University's central Exchange email service, experienced a disruption. The service was restarted and fully restored by approximately 8:30 a.m. ITS staff are investigating the software failure which caused the disruption and will take steps to ensure it does not happen again.
01-30-2012 8:07 PM Green Services Restored
Jan. 30, 2012 - As of 8 p.m. all services are fully restored. Recap of event: At approximately 6:45 p.m. Jan. 30, a network failure in the 6045 Data Center caused a disruption in several services, including Maximo, CBORD, News, and Webshare, among others. Most services were restored by 7:05 p.m. and all were restored by 8 p.m. IT Services will be escalating this case with our network vendor to determine the nature of the failure and a permanent fix.
01-19-2012 8:34 AM Green xMail Slowness - Resolution Underway
Jan. 19, 2012 - We have begun a mitigation process for the xMail slowness by moving some people off to other servers. The process will continue over the weekend; the slowness issues should resolve as the migration continues. A permanent resolution will be forthcoming in the next few months as we migrate xMail to a new Exchange environment. Recap of event: Over the past few days, users of xMail, the University's central Exchange-based email and calendaring service, have been experiencing sporadic issues with slow or delayed response times. This is because our Exchange infrastructure is nearing its maximum capacity in terms of memory usage, causing occasional slow response times during peak usage. We are mitigating this by distributing users across more servers.
12-27-2011 9:35 AM Green RESOLVED: VoIP Call Center Issue Affecting IT Services Contacts
Dec. 27, 2011 - At approximately 8:30 a.m. an issue was detected with the portion of the VoIP phone system which provides call center functionality. The only client utilizing the VoIP call center function which was impacted is IT Services. As a result, attempts to reach IT Services via 2-5800, 2-9100, 4-8324 or 2-1234, and reach a live contact, were queued on hold until 9:10 a.m. Service was restored at approximately 9:10 a.m. and calls are now being answered normally. ITS will investigate the cause of call center issue to see how it can be prevented in the future.
12-14-2011 9:47 AM Green RESOLVED - Authentication Issues
Dec. 14, 2011 - As of 9:20 a.m., login problems to ITS services have been resolved. Recap of event: At approximately 9 a.m., the primary campus authentication servers (LDAP) ran out of memory following a software upgrade. As a result, people encountered problems logging on to servers depending on this authentication, including wireless, cMail, Chalk, VPN, and several administrative systems (UCTime, GEMS, etc.), among others. The issue was resolved at approximately 9:20 a.m. by reverting to the old software version and restarting the services. ITS will investigate the cause of the memory issue before re-attempting the upgrade at a later date.
12-01-2011 7:36 AM Green RESOLVED: Wireless on North Side of Campus
Dec. 1, 2011 - As of approximately 2:40 a.m. this morning, wireless on campus is again functioning normally. Recap of event: At approximately 9:30 a.m. Wednesday, Nov. 30, a configuration issue began causing sporadic network loops between two wireless gateways. As a result, users of the non-secure 'uchicago' network began seeing sporadic network disruptions, as the gateways reset themselves to break the loop. The issue was resolved at approximately 2:40 a.m. Dec. 1 by reconfiguring the affected network links. The secure wireless networks (uchicago-secure and eduroam) were not affected by this problem.
11-16-2011 10:17 PM Green RESOLVED: Email Bouncing Issues
Nov. 16, 2011 - As of approximately 9:15 p.m., email and authentication services are again operating normally. IT Services staff will be contacting individuals who had email returned, so they can contact their correspondents and ask for a re-send. Recap of event: At approximately 2 p.m., an issue on the University's central authentication servers (LDAP) began delaying updates for password changes, email forwarding changes, and new account creation. At approximately 8:20 p.m., the problem expanded to existing email forwarding. As a result, roughly 4% of our email traffic (~550 messages) were incorrectly returned to the original senders. Normal operations were restored by 9:15 p.m. IT Services staff will be contacting people whose email bounced, and will also be investigating the cause of the issue to see how it can be prevented in the future.
11-15-2011 1:25 PM Green RESTORED: Wireless Authentication Issue
Nov. 15, 2011 - At approximately 12:30 p.m., a memory issue on the wireless authentication server began causing connection problems for users across campus. IT Services staff were alerted to the problem and fixed the issue by approximately 1:05 p.m. If you are still having problems connecting, try restarting your machine, or call the Service Desk at 2-5800 (773-702-5800). IT Services staff will be re-tuning the authentication server to help prevent this error in the future.
11-01-2011 7:02 PM Green RESTORED: cMail/Webmail
Nov. 1, 2011 - At approximately 6:40 p.m., webmail.uchicago.edu, the web front end to cMail, stopped responding to requests. The problem was fixed by restarting the webmail servers; service was fully restored by 6:50 p.m. IT Services staff will be working with the vendor to determine how this issue can be prevented in the future.
10-25-2011 4:07 PM Green Chalk File Services Disruption
Oct. 25, 2011 - At approximately 3:45 p.m., a database issue caused file services on Chalk, the University's central learning management system, to become unavailable. IT Services staffed tracked down the problem and corrected it. File services were fully restored by 4:05 p.m.
10-03-2011 1:35 PM Green Email Delivery Restored
As of 1:30 PM, email delivery is operating normally. No email was lost, but it may take up to four hours before all the backlogged email is delivered. Recap of event: On the morning of Monday, October 3rd, a routine change to our antispam configuration triggered a bug in the antispam service. This bug effectively stopped email delivery to @uchicago.edu and @lists.uchicago.edu as of approximately 10:45 AM October 3rd. IT Services staff were able to work with the vendor to isolate and reverse the change; some messages began flowing through at approximately 1 PM, and by 1:30 PM, the service was fully restored. IT Services will continue working with the vendor to find a long-term solution.
09-23-2011 4:00 PM Green Email attachment delivery problem resolved
As of 4 PM, email delivery service on xMail is operating normally. IT Services will recover the removed attachments from quarantine and deliver them to their destination addresses. Please note that you may receive multiple copies of an attachment if multiple copies were sent during the outage. Recap of event: At approximately 2:50 PM Friday, September 23rd, the antivirus software for xMail, the University's central Exchange email service, began to erroneously reject a large number of attachments when attempting to scan them for viruses. The body of the message was still being delivered, and messages without attachments were unaffected; only attachments were being removed. IT Services staff traced the issue to queued messages from this morning's disruption which had not processed correctly, and cause a recurrence of this morning's issue. They were able to resolve the issue by manually clearing that queue. Email service was fully restored by 4 PM.
09-23-2011 10:28 AM Green Email attachment delivery problem resolved
As of 10:30 AM, email delivery service on xMail is operating normally. IT Services will recover the removed attachments from quarantine and deliver them to their destination addresses. Please note that you may receive multiple copies of an attachment if multiple copies were sent during the outage. Recap of event: At approximately 8:45 AM Friday, September 23rd, the antivirus software for xMail, the University's central Exchange email service, began to erroneously reject a large number of attachments when attempting to scan them for viruses. The body of the message was still being delivered, and messages without attachments were unaffected; only attachments were being removed. IT Services staff traced the issue to a failed automated update on the antivirus engine, and were able to resolve the issue by re-applying the update. Email service was fully restored by 10:30 AM.
08-30-2011 3:22 PM Green Brief Service Disruption for cMore, other websites
At approximately 2:07 PM Tuesday, August 30th, a number of IT Services websites began reporting errors. Sites affected included cMore, Society of Fellows, Student Employment, Graham School, Career Planner, Events calendar, and Faculty Access. The error was traced to a faulty code push and service was restored by 2:22 PM. IT Services will be reviewing the process to ensure this does not happen again.
08-02-2011 12:57 PM Green Mailing lists restored
As of 12:50 PM, mailing list service has been restored and all delayed email has been delivered. The service is now operating normally. Recap of event: At approximately 12:20 PM Tuesday, 02 August, the University's central mailing list server (lists.uchicago.edu) encountered a storage problem which caused it to stop forwarding email. No email was lost, but delivery of email sent to mailing lists was delayed IT Services staff were able to restart the service and remount the storage device, restoring service by 12:50 PM. IT Services will continue to monitor the service closely and take steps to prevent this from happening again.
07-21-2011 5:00 PM Green Email attachment delivery problem resolved
As of 4:45 PM, email delivery service on xMail is operating normally. IT Services will recover the removed attachments from quarantine and deliver them to their destination addresses. Recap of event: At approximately 3:25 PM Thursday, 21 July, the antivirus software for xMail, the University's central Exchange email service, began to erroneously reject a large number of attachments when attempting to scan them for viruses. The body of the message was still being delivered, and messages without attachments were unaffected; only attachments were being removed. IT Services staff traced the issue to a failed automated update on the antivirus engine, and were able to resolve the issue by re-applying the update. Email service was fully restored by 4:45 PM.
07-18-2011 3:20 PM Green Employee Self-Service Disruption
At approximately 2:30 PM Monday, 18 July, the Employee Self-Service (ESS) website stopped responding. The service was restored at approximately 3 PM by restarting the web service. IT Services staff are continuing to monitor the service to ensure its continued stability.
07-15-2011 1:14 PM Green Student ID Cards re-enabled
As of 1 PM, 15 July, valid student ID cards should be working normally. If you are still having issues with your ID card, contact the ID and Privileges Office on the first floor of the Regenstein Library (JRL100F; 773-702-3344). IT Services staff continue to monitor the system for issues, and will work to isolate and fix the original cause of the error. Recap of event: On the morning of 15 July, an error occurred during a standard student status update which caused 11,000 student ID cards to be erroneously marked as inactive. Due to the timing of the update, valid student ID cards could still be used to access Library and Athletics facilities, but students might have been unable to use their cards for door access on other parts of campus. IT Services staff corrected the issue by pulling in a full correct student update and propagating it to door access systems. IT Services staff continue to monitor the system for issues, and will work to isolate and fix the original cause of the error.
06-30-2011 1:45 AM Green Mailing list service restored
As of 1:30 AM, 30 June, the campus mailing list server (lists.uchicago.edu) is processing email normally. There is a backlog of email which should be delivered by 2 AM. Approximately 225 messages were bounced back to the senders with spurious errors (notably "no such list exists"); IT Services staff will attempt to contact senders and list owners to let them know that these messages can be re-sent. Recap of event: At approximately 8:23 PM Wednesday, June 29th, the campus mailing list server (lists.uchicago.edu) experienced a problem with its data storage; this caused it to begin queuing (or, in some cases, spuriously rejecting) messages instead of delivering them. IT Services engineers were able to reconnect the data storage at approximately 1:15 AM Thursday, June 30th, and begin processing queued messages. The system will be monitored closely through the night to ensure that the issue is resolved, and a planned upgrade to the system will be fast-tracked.
06-23-2011 11:27 AM Green Campus Phone Disruption - Resolved

As of 11:25AM, normal phone services have been restored across campus. At approximately 11:00AM IT Services engineers migrated campus phones off the affected Hospital Call Manager. Migrated phones have been tested and we have confirmed that the earlier problems have been eliminated. If you are still having issues with your phone, please contact us for further assistance via email at 29100@uchicago.edu or voice (from another working phone) at 773-702-9100, option 1.

Recap of event: Shortly after 8 AM, Thursday, 23 June, the campus Voice over IP (VoIP) telephones began experiencing intermittent problems. Symptoms that VOIP phones may experience include, a loss of dialtone and/or one-way audio. IT Services engineers are working to restore service and determine what is causing this issue.

06-21-2011 9:49 AM Green Young Building Restored
Network service in Young has been restored. Recap of event: At approximately 9:30 AM Tuesday, 21 June, electrical work in Young caused a network switch to briefly lose power. As a result, the building lost network and phone connectivity. The switch was rebooted and service restored by 9:33 AM.
06-14-2011 11:36 PM Green Campus Phone Service Restored
As of 11:30 PM, phone service on campus has been fully restored. IT Services staff will investigate the cause for deletion and will revisit processes to ensure this does not happen again. Recap of event: Shortly after 7 PM, Tuesday, 14 June, approximately 1700 phones were incorrectly deleted from one of the central Voice over IP (VoIP) call managers. Calls to the deleted phone numbers (which are scattered across campus) are currently not going through. IT Services engineers restored the database from backup. Service was fully restored by 11:30 PM.
06-08-2011 1:39 PM Green Email delivery problem resolved
At approximately 10:30 AM Wednesday, 08 June, an error began occurring in the process that identifies who has the right to forward email from @uchicago.edu. Over the next two hours, a large portion of the uchicago.edu alumni community was marked as not being eligible for forwarding; as a result, email to these people was returned to its senders. As of 1:30 PM, the permissions have all been corrected, and email is being delivered normally. IT Services is working to find out what led to the error and put in a permanent fix.
06-08-2011 6:29 AM Green Tank service restored
At approximately 4:30 PM Tuesday, 07 June, the tank.uchicago.edu file sharing service experienced an apparent memory hardware failure. As a result, Tank users began having sporadic and unpredictable connection issues. As of 6:15 AM Wednesday, 08 June, the vendor has applied a software patch to address a bug which was the source of the original error; the system has passed all health checks and we believe the service is restored. Tank users may need to restart their machines to re-associate with the fileshare; if you are still having issues after rebooting, please contact the Service Desk at 773-834-TECH.
06-07-2011 10:02 PM Green Tank Filesharing Service Restored
At approximately 4:30 PM Tuesday, 07 June, the tank.uchicago.edu file sharing service experienced an apparent memory hardware failure. As a result, Tank users began having sporadic and unpredictable connection issues. As of 10 PM, we believe that the service is restored. IT Services is continuing to work with the vendor to discover why failover did not work correctly.
06-02-2011 6:46 PM Green Account Claiming Service Restored
As of 6:30 pm on Thursday, June 2, 2011, IT Services has resolved the error condition which disrupted access to the account claiming service. We will be monitoring the service closely to detect and resolve any recurrence.
05-26-2011 11:13 PM Green Mailing List Service Operating Normally
As of 11:12 PM on Thursday, May 26, the mailing list service has been restored to full operation. IT Services engineers are reviewing the cause of the problem and will take appropriate measures to prevent recurrence.
04-26-2011 3:01 PM Green uchicago Wireless on North-side of Campus Resolved
The problem affecting the wireless network on the north-side of campus (roughly north of 58th St), has been resolved. Clients can once again use the legacy captive-portal "uchicago" wireless network. IT Services has modified the configuration of the network to prevent this issue from recurring.
04-26-2011 1:34 PM Green uchicago Wireless on North-side of Campus Restored
The problem affecting the wireless network on the north-side of campus (roughly north of 58th St), has been bypassed. Clients can once again use the legacy captive-portal "uchicago" wireless network. IT Services staff continue to work on the underlying cause of the issue.
03-30-2011 11:33 AM Green Access Into junkmail.uchicago.edu Restored
As of 9:50AM on Wednesday, March 30, all users should now have ready access into their spam quarantine and all spam filtering levels have returned to normal. IT Services is working with vendor partners to analyze the source of this issue and prevent it from recurring.
03-30-2011 8:58 AM Green Mail Delivery Rate Normal
As of 9:00 AM on Wednesday, March 30, we have completed processing the backlog of mail which accumulated this morning. All mail deliveries should be occurring normally.
03-18-2011 10:26 AM Green cMail Restored
At approximately 10:10 AM on Friday, 18 March, one of the back end mail servers for cMail began experiencing issues. As a result, approximately one-fourth of the people using cMail via the web interface lost access to their mail. Service was restored at approximately 10:25 AM. IT Services staff are actively investigating the source of the problem with vendor partners.
03-17-2011 10:35 AM Green Brief xMail Service Interruption
At approximately 10AM on 3/17/2011, a process running on the xMail servers began consuming excessive resources, resulting in a brief period where clients may have disconnected or been asked to log in again. The process is no longer consuming excessive resources and the xMail server has resumed normal operation. IT Services staff continue to monitor the xMail service.
02-25-2011 3:24 PM Green cMail Restored
At approximately 3:10 PM on Friday, 25 February, one of the back end mail servers for cMail began experiencing issues. As a result, approximately one-fourth of the people using cMail via the web interface lost access to their mail. Service was restored at approximately 3:20 PM. IT Services staff are actively investigating the source of the problem with vendor partners.
02-21-2011 9:03 AM Green cMail Outage Resolved
As of 8:55AM on 21 February, the condition which precipitated the earlier cMail outage has been resolved and all logins to the cMail system are now working properly. Clients initially reported an inability to delete email via the webmail interface shortly after regaining access at 8:30AM, and this issue has also been resolved. IT Services staff are actively investigating the cause of this deletion issue and are in contact with the vendor.
02-17-2011 1:58 PM Green cMail Service Restored
As of 1:55PM on 17 February, the condition which precipitated the earlier cMail outage/slowdown has been resolved. Logins to the affected portion of the cMail system should now be occurring normally. IT Services staff are actively investigating the cause of the slowdown and are in contact with the vendor about the issue.
02-07-2011 9:58 AM Green Chalk available again
At approximately 8:55 AM Monday, 07 February, a module configuration error caused the main page in Chalk to render incorrectly. The error was traced down and corrected at approximately 9:45; Chalk is now operating normally. IT Services staff is tracking down the source of the error and will take steps to prevent it happening again.
02-03-2011 9:25 PM Green Services return to normal on Friday, February 4
IT Services returns to normal operations on Friday, February 4.
(Read More...)
01-18-2011 8:54 AM Green Partial cMail Outage Resolved
As of 8:50AM on 18 January, the condition which precipitated this morning's partial cMail outage has been identified and resolved. Logins to the affected portion of the cMail system should now be occurring normally. IT Services staff continue to monitor the cMail system and are in contact with the vendor about the issue.
01-08-2011 12:57 PM Green xMail Service Fully Available
IT Services engineers have restored the xMail service to full availability and are investigating the cause of the problem. We will continue to monitor the service closely throughout the day.
12-22-2010 12:13 PM Green Webshare and Proxy now available.
IT Services engineers have applied a workaround to restore access to Webshare and the campus proxy (proxy.uchicago.edu and associated academic resources accessed through it). IT Services continues to monitor all services and the larger network to ensure continued stability.
12-22-2010 10:25 AM Yellow Webshare and Proxy Unavailable
As a consequence of network problems during maintenance this morning, webshare.uchicago.edu and proxy.uchicago.edu, which provides access to a range of protected academic resources, are currently unavailable. IT Services engineers are working with vendor partners to address this issue as quickly as possible.
12-22-2010 8:12 AM Green Resolved: Web, cMail, multiple services impacted by maintenance-related outage.
On the morning of December 22nd, IT Services personnel ran into some unforeseen difficulties while conducting maintenance on some core network components in the data center. Due to these difficulties, a number of core web services including Chalk, www.uchicago.edu, hosted databases, and web-based applications, became unavailable at 5:45am. IT Services personnel quickly recognized the origin of the issue and have corrected it as of 7:30am.
12-11-2010 12:47 PM Green Partial cMail Outage Resolved
As of 12:15PM on 12/11/2010, the condition which precipitated yesterday's partial cMail outage has been identified and resolved. Logins to the affected portion of the cMail system should now be occurring normally. IT Services staff continue to monitor the cMail system and are contacting the vendor about the issue.
12-11-2010 12:12 PM Yellow Partial cMail Outage
At approximately 7:00PM on Friday, 10 December, one of the mail storage units on cMail, the University's central email system, began experiencing issues. As a result, approximately one-fourth of the people using cMail have currently lost access to their mail. IT Services staff have been notified of the issue and are looking into the problem.
12-07-2010 9:55 AM Green College Registration Back Up
On Monday, 6 December, shortly after opening for Winter 2011 Add/Drops at 9AM, the college-registration site began generating excessive load on a portion of our web infrastructure. The excessive load began to affect other sites and at approximately 9:40AM, IT Services closed the college-registration site. After performance testing and installing some new controls, college-registration was reopened at 9 AM on December 7th, and appears to be stable. IT Services is investigating capacity improvements for future quarters.
12-06-2010 9:59 AM Yellow College-registration Site Temporarily Closed
Shortly after opening for Winter 2011 Add/Drops at 9AM, the college-registration site began generating excessive load on a portion of our web infrastructure. The excessive load began to affect other sites and at approximately 9:40AM, IT Services temporarily closed the college-registration site. IT Services staff are working on the issue and we will post updates as they develop.
12-01-2010 10:16 AM Green Brief Network Disruption
Between 9:45 and 9:55 AM Wednesday, 01 December, the campus network briefly lost connectivity with the outside world. IT Services network engineers are investigating the cause of the disruption.
11-20-2010 12:48 AM Green All Services Now Available
At approximately 7:45 PM Friday, 19 Nov, a failure on our central data storage devices made several services unavailable. Affected services included Webshare, My uchicago, cTicket, and the cAlert sign-up page, among others. As of 12:30 am on Saturday, 20 Nov, all services have been restored. Please note, this incident did NOT affect emergency notification ability, only the ability to sign up for cAlert notifications. IT Services has confirmed that all mail was correctly handled and delivered during the course of this event. Staff will continue to monitor services closely and work with our vendors to trace the cause of the problem we experienced.
11-20-2010 12:07 AM Yellow Most services restored; cAlert Sign-on Page still down
At approximately 7:45 PM Friday, 19 Nov, a failure on our central data storage devices made several services unavailable. Affected services include Webshare, My uchicago, cTicket, authenticated SMTP, and the cAlert sign-up page, among others. As of 12 midnight Saturday, 20 Nov, all services have been restored except the cAlert sign-up page. This does NOT affect emergency notification ability, only the ability to sign up for cAlert notifications. IT Services staff are working with vendor support to finish restoring this service. Additionally, it is possible that some outgoing email may have been lost between 7:45 and 10:15 PM; IT Services is working to identify any lost email and will notify senders to the degree possible.
11-19-2010 9:30 PM Red Webshare, My Uchicago outage
At approximately 7:45 PM Friday, 19 Nov, a failure on our central data storage devices made several services unavailable. Affected services include Webshare, My uchicago, cTicket, and the cAlert sign-up page, among others. As of 9:15 PM, IT Services and vendor support are working to restore service as quickly as possible, but we anticipate that services will not be completely restored before 11 PM or later.
11-19-2010 8:12 PM Red Outage of Portal, other services
At approximately 8 PM Friday, 19 Nov, a failure on our central data storage devices made several services unavailable. Affected services include My.uchicago (the University portal) and WebShare, among others. IT Services staff are working to restore service.
11-18-2010 7:46 AM Green ARD Virtual Desktop Infrastructure Available
As of 4:20am on Thursday, November 18, the VDI pool for Alumni Relations and Development is enabled and online for use. IT Services engineers are working with our vendor partners to analyze the cause of this issue and prevent a recurrence.
11-18-2010 12:55 AM Yellow ARD Virtual Desktop Infrastructure currently unavailable
On Thursday, November 18 at 12:45am, IT Services disabled access into the Alumni Relations & Development VDI environment in order to troubleshoot a series of error messages and login issues. IT Services engineers and the vendor are actively working to resolve this issue.
11-03-2010 4:31 PM Green cMail Service Interruption
On Wednesday, November 3, at 3:47 pm, some users experienced errors when using the webmail (cMail) service. IT Services staff addressed the problem and as of 3:55 pm, the service is now operating normally for all users. We are following up with the vendor to prevent the problem from recurring.
11-03-2010 9:32 AM Green Outlook Certificate Error
Wednesday, November 3. Due to an upgrade of the xMail system during our regularly scheduled maintenance period, a small percentage of Outlook users may see a certificate error. Restarting Outlook, or choosing "Yes" when asked "Do you want to proceed?" should clear the error permanently. Please contact 773-834-TECH (8437) or support@uchicago.edu if you have questions or concerns.
11-03-2010 9:28 AM Green Outlook Certificate Error
Wednesday, November 3. Due to an upgrade of the xMail system during our regularly scheduled maintenance period, a small percentage of Outlook users may see a certificate error. Restarting Outlook, or choosing "Yes" when prompted to "Trust This Certificate" should clear the error permanently. Please contact 773-834-TECH (8437) or support@uchicago.edu if you have questions or concerns.
10-20-2010 7:15 AM Green Webshare Maintenance Complete
The scheduled maintenance and upgrades to Webshare were completed successfully by 6:00am on Wednesday, October 20. The service is now available for use.
10-19-2010 9:35 PM Yellow Scheduled Webshare Maintenance
Webshare.uchicago.edu will be unavailable due to scheduled maintenance and upgrades starting at 5:00am on Wednesday, October 20. IT Services expects to complete this activity by 7:00am.
10-18-2010 12:25 PM Green cTV service missing channels - resolved
IT Services has corrected the configuration issue which was preventing clients from viewing channels in certain areas. Access to all channels was restored at approximately 12:05PM. We apologize for any inconvenience this configuration issue caused.
10-18-2010 11:54 AM Yellow cTV Service missing channels
IT Services has identified a configuration issue with the infrastructure providing connectivity for the cTV service. This configuration issue is causing an inability for clients (both the java-based web client and physical set-top boxes) to view certain channels in many locations - including, but not limited to channels 2, 7, 9, 11, 32, 100, 101, 150 and 170. We will be correcting the configuration issue beginning at approximately Noon on Monday, 10/18/2010. We expect to complete the fix, restoring access to all channels, by 1:00PM. We apologize for any inconvenience this configuration issue has caused.
10-05-2010 12:03 PM Green xMail Certificate Issue Resolved
At approximately 11 AM, Tuesday, 5 October, a problem with a new security certificate began causing misleading "untrusted certificate" or "untrusted connection" errors for some xMail users on a variety of email clients, including Outlook, iPhone, and web-based email. As of 11:50 AM, the problem certificate has been removed and the original certificate restored; xMail should now be working normally.
10-05-2010 11:13 AM Yellow xMail Certificate Warnings
A problem with a new security certificate is causing some xMail users to receive error messages about an "untrusted certificate" or "untrusted connection" when checking their email. These errors occur on Outlook, iPhones, xmail.uchicago.edu, and other clients, and can be bypassed by clicking "Yes" and/or accepting the certificate. IT Services is working to resolve the issue.
09-29-2010 9:53 AM Green Connectivity Restored
At approximately 6 AM Wednesday, 29 September, a denial-of-service attack began causing intermittent network problems on campus. This included slow performance and occasional loss of connection between the campus network and the outside world. As of 9:30 AM, the source of the attack has been isolated, and normal network performance has been restored.
09-29-2010 9:12 AM Red Severe Connectivity Issues
At approximately 8:56AM on 9/29/2010, connections both to and from campus to the internet began experiencing severe performance issues. IT Services staff are working to resolve the issues.
09-29-2010 8:50 AM Yellow Intermittent Network Connectivity Problems
Some parts of the campus network are experiencing intermittent connectivity problems and/or slowness as of approximately 6AM. IT Services Network Engineers are actively investigating these problems. We will post an update when more information is available and provide regular status updates.
09-08-2010 2:47 PM Green SAN operating normally
At approximately 2:40pm on Wednesday, 9/8/2010, the problematic node of our Storage Area Network (SAN) was returned to service. SAN operation has returned to normal and IT Services staff will continue to monitor the SAN closely.
09-08-2010 2:00 PM Yellow SAN issue potentially impacting performance of services
At approximately 1:30pm on Wednesday, 9/8/2010, one of the redundant nodes providing connectivity for our Storage Area Network (SAN) went down, resulting in performance degradation for approximately one-half of the storage environment. Although no services are unavailable at this time, services that utilize the SAN may exhibit some performance issues while we work to resolve this situation. IT Services staff are working with the vendor on resolving this issue.
09-07-2010 12:12 AM Green Resolved: WebShare (webshare.uchicago.edu) offline.
As of 11:45 pm on Monday, September 6th, the WebShare service is now fully available and ready for use. IT Services' assessment of the system revealed that no files were lost, but a small number of files which were uploaded at the time the error occurred may not be appearing correctly. We have contacted the owners of these files. IT Services staff will monitor the system closely to identify any further issues.
09-06-2010 10:14 PM Yellow Update: WebShare (webshare.uchicago.edu) offline.
At 8:00 pm on Monday, September 6, an error condition in the campus storage area network (SAN) caused corruption in the WebShare database. The WebShare service is currently offline while IT Services staff perform a series of data restores and health checks.
09-06-2010 9:27 PM Yellow WebShare (webshare.uchicago.edu) offline.
At around 8:45 pm on Monday, September 6, the WebShare service (http://webshare.uchicago.edu/) became unavailable. IT Services staff is working to bring the service back online as soon as possible.
09-05-2010 8:52 AM Green Network Maintenance Complete
As of 2 a.m. (U.S. Central Time) on Sunday, September 5, maintenance on the data network is complete and all University systems are available for use.
09-03-2010 3:34 PM Yellow Scheduled Network Maintenance
From Saturday, September 4 at 11 p.m. to Sunday, September 5 at 2 a.m. (U.S. Central Time), IT Services will be performing maintenance on the data network. We will attempt to minimize the impact of this maintenance, but there will be brief periods when some University systems, including some voice systems, will be unavailable.
09-02-2010 3:27 PM Green cMore operating normally
After closely monitoring the performance of cMore this afternoon, IT Services staff are no longer seeing evidence of performance problems. The Gargoyle administrative application is also operating normally at this time. We will continue to monitor both cMore and Gargoyle closely to prevent recurrence of this issue. Thank you for your patience with this situation.
09-02-2010 1:13 PM Yellow cMore re-opened
IT Services Staff have re-opened cMore as of 1:05pm on 9/2/10. The Gargoyle Administrative interface is also now available to staff users, as well. We will continue to monitor performance issues throughout the rest of the day and may take additional action to maintain acceptable performance levels.
09-02-2010 11:56 AM Red cMore temporarily closed
IT Services staff have temporarily closed cMore while we work on resolving the performance issues which began at 10:30am on Thursday, 9/2/10. Performance of sites - other than cMore - hosted on the University's main web servers, should be back to normal. We currently have no estimate for when cMore will be reopened.
09-02-2010 11:37 AM Yellow Performance issues affecting the university's main web servers (update)
At approximately 10:30 am on Thursday, 9/2/10, performance of the University's main web servers was degraded due to high traffic. IT Services staff have identified cMore as the site receiving the spike in traffic, which resulted in cMore becoming unavailable at times and other sites becoming slow. IT Services staff are reallocating web resources to better handle the cMore load, which should reduce the overall impact on the web servers. We will continue to monitor and adjust as the high traffic situation evolves.
09-02-2010 10:56 AM Yellow Performance issues affecting the university's main web servers
Performance of the University's main web servers is currently degraded and some web sites may respond slowly or not at all. IT Services is working to investigate and resolve the problem. We currently have no estimate for when full performance will be restored.
08-25-2010 8:14 AM Green Wireless Network Maintenance Complete
As of 8 a.m. (U.S. Central Time) on Wednesday, August 25, maintenance on the wireless network is complete and the wireless network is available for use.
08-24-2010 5:01 PM Yellow Wireless Network Maintenance, August 25
On August 25, 2010 from 12:00 a.m. to 7:00 a.m. (U.S. Central Time), IT Services will be performing maintenance on the wireless network. We will attempt to minimize the impact of this maintenance, but there will be sporadic periods when the wireless network will not be available.
08-24-2010 6:15 AM Green cMail Maintenance Complete
As of 6:00am on Tuesday, August 24th, we have completed our maintenance on the cMail system. We believe that we have resolved all of the critical issues with the new mail interface, and will respond to any new issues as they arise.
08-23-2010 5:31 PM Yellow IT Services Emergency cMail Maintenance
At 4:00 am on Tuesday, August 24th, we will be performing emergency maintenance on the cMail service. This maintenance will correct multiple critical problems with the webmail interface, including an issue with deleting messages. Please be aware that this emergency maintenance will cause a momentary outage in the webmail interface. If you are in the midst of composing a message when the maintenance starts, you will lose your unsaved message, but you will not lose any other messages.
08-23-2010 10:25 AM Yellow cMail message deletion issue.
For people using the webmail interface for cMail, the preference of marking messages for deletion is not working at this time. IT Services is working with vendor support to resolve this problem. In the meantime, to work around this issue, unmark any messages marked for deletion, click "Preferences" in the left sidebar, turn on "Delete to Trash" and click "OK." When you delete messages, it will send those messages to the Trash folder. You can click on the "Empty" button to permanently remove messages from the Trash folder. Step-by-step instructions are available at http://itservices.uchicago.edu/support/
08-22-2010 9:20 AM Green Mail System Maintenance Completed
Maintenance on the university mail system (xMail, cMail, Junkmail) completed today at 9:15 am. All systems continue to be available for use. Some cMail IMAP users may experience a brief delay during their first login to the updated system. IT Services staff will continue to monitor the situation throughout the day.
08-20-2010 3:29 PM Yellow Scheduled Email Maintenance, August 22
IT Services is currently conducting scheduled maintenance on the University email environment (cMail, xMail, Junkmail). This maintenance is now expected to complete by 9:30am. During the maintenance period, all services will continue to be available, but you may experience short email delays or email slowness.
08-19-2010 5:54 PM Green Email Delivery Problem Resolved
On the afternoon of Thursday, August 19th, problems on our incoming mail servers caused email delivery to be delayed and, in some cases, possibly erroneously returned to the sender. As of 5:30 PM, the backlog has cleared and email delivery is functioning normally. IT Services staff continue to work with the vendor to identify the exact cause of the problem and take steps to prevent any recurrence.
08-19-2010 4:37 PM Yellow Mail Delivery Returning to Normal Speed
We believe that the earlier problem with incoming mail delivery to @uchicago.edu addresses has been resolved. IT Services administrators will continue to monitor email flow to ensure the problem does not recur. Some emails may be slightly delayed as backlogs are cleared.
08-19-2010 3:20 PM Red Mail Delivery Issues
Incoming mail delivery to @uchicago.edu addresses is currently experiencing problems, which is causing some mail to be returned to the sender erroneously. IT Services System Engineers are working with our mail system vendor to resolve the issue.
08-14-2010 12:17 AM Green UPS Maintenance Complete
IT Services has completed maintenance on the Uninterruptible Power Supply (UPS). We have successfully corrected the error condition and verified the status of affected systems. All applications are now believed to be operating normally. Technical staff in IT Services will monitor the UPS to verify that no additional issues arise. If you encounter any problems, contact IT Services Support at 834-TECH or support@uchicago.edu.
08-13-2010 8:19 PM Yellow UPS Maintenance In Progress
As of 8:20 pm on Friday, August 13th, maintenance on the Uninterruptible Power Supply (UPS) for the main campus data center is currently underway and on-schedule. For more information about service availability during the maintenance period, see http://itservices.uchicago.edu/ups. If you have questions, please contact (773) 834-TECH or support@uchicago.edu.
08-13-2010 7:13 PM Yellow Maintenance Preparations Continue
As of 7:00 pm on Friday, August 13th, UPS Maintenance preparation activities are continuing, with all advance shutdowns proceeding on schedule. For more information about service availability during the maintenance period, see http://itservices.uchicago.edu/ups. If you have questions, please contact (773) 834-TECH or support@uchicago.edu.
08-13-2010 6:09 PM Yellow Protective Service Shutdowns Continue
As of 6:00 pm on Friday, August 13th, IT Services has begun a planned series of service shutdowns for the duration of the maintenance on the campus data center UPS. For more information about service availability during the maintenance period, see http://itservices.uchicago.edu/ups. If you have questions, please contact (773) 834-TECH or support@uchicago.edu.
08-13-2010 3:21 PM Yellow Service Preparation for UPS Maintenance Has Begun
As of 3:00 pm on Friday, August 13th, service preparation for the UPS maintenance in the main campus data center has now begun. As part of the operational plan, the mainframe online regions are now unavailable to allow end of day batch processing to begin. Additional protective service shutdowns are slated to begin at 6:00 pm. For more information about service availability during the maintenance period, see http://itservices.uchicago.edu/ups.
08-11-2010 10:23 PM Green All sites now available
As of 9:15 pm on Wednesday, August 11, we believe all systems have been restored. If you experience any service disruptions, please contact support@uchicago.edu or 4-TECH.
08-11-2010 6:58 PM Yellow All Sites Now Available; Some Repairs Continue
As of 6:42 pm, on Wednesday, August 11, IT Services has restored access to all sites. In addition, some redundant and failover systems are not yet returned to service. Some portal users are reporting system slowness, and new events cannot be added to the campus events calendar.
08-11-2010 6:30 PM Red Campus Portal, Some Departmental Sites Unavailable
At 5:03 pm, on Wednesday, August 11, the campus portal, the campus events calendar, and several departmental websites became unavailable. IT Services is aware of the problem and is actively working to bring these systems back online.
08-11-2010 1:41 PM Green Intermittent Problems with cMail, DNS, LDAP, Lists and Wiki Resolved

At approximately 10:54AM on Wednesday, August 11, a load balancing component of our infrastructure unexpectedly failed and its load was transferred to its redundant peer. Various services located behind the load balancers - including cMail, DNS, LDAP, Lists and Wiki - experienced intermittent problems and were unavailable at times until just after Noon. All services are no operating normally.

A moderate backlog of email developed while this problem occured, resulting some mail delivery being delayed by up to an hour. As of 1PM all queued mail has been delivered and normal delivery has resumed.

08-11-2010 11:46 AM Yellow Intermittent Problems with cMail, DNS, LDAP, Lists and Wiki

At approximately 10:54AM on Wednesday, August 11, a load balancing component of our infrastructure unexpectedly failed and its load was transferred to its redundant peer. Various services located behind the load balancers - including cMail, DNS, LDAP, Lists and Wiki - are currently experiencing intermittent problems and may be unavailable at times. Symptoms that might be encountered range from an inability to access a service at times, intermittent timeouts and/or authentication failures.

IT Services systems staff are currently working to resolve the problem.

08-11-2010 7:10 AM Green Scheduled Email Maintenance Complete
The email maintenance is complete for August 11th. Email routers have been upgraded to new hardware and the latest software. IT Services will be monitoring throughout the day. All email systems are available for use.
(Read More...)
08-10-2010 1:23 PM Yellow Scheduled Email Maintenance, August 11
IT Services will be conducting scheduled maintenance on the University email environment (cMail, xMail, Junkmail) on August 11, 22, and 25 from 4 a.m. to 7 a.m. (U.S. Central Time). During these three regular Wednesday and Sunday maintenance periods, you may experience short email delays or email slowness.
08-08-2010 11:12 AM Green Telephone service restored
At approximately 7:30 PM Saturday, August 7th, regularly scheduled ComEd maintenance caused a power outage on the south side of campus, including part of the campus telephone service. This service ran for a while on battery power, but when the battery ran out at approximately 9:15 PM, non-VOIP telephone service for several buildings on the south side of the quads was lost. Electrical power was restored at approximately 10:20 AM Sunday, August 7th, and phone service was restored shortly after. At this point, all services should be operating normally.
08-07-2010 8:30 PM Red Telephone Service Outage on South Side of Quads
At approximately 7:30 PM Saturday, August 7th, regularly scheduled ComEd maintenance caused a power outage on the south side of campus, including part of the campus telephone service. This service is currently running on battery power; when that runs out at approximately 9:15 PM, non-VOIP telephone service for several buildings on the south side of the quads will be lost. Data service should not be affected. ComEd and Facilities have assured us that the problem will be fixed no later than 6 AM. Buildings affected include Rosenwald, Walker, Pick, Kelly, Beecher, Green, Foster, Social Science, Harper, Stuart, Wieboldt, Classics, Goodspeed, Gates Blake, Cobb, Swift, Oriential Institute, Haskell, and Presidents House.
07-07-2010 11:20 AM Green Resolved: cTV: partial service interruption
IT Services staff, working in conjunction with the vendor, restored all missing cTV channels as of 11:10am on July 7th. cTV is now operating normally and all maintenance is complete. IT Services staff will continue monitoring cTV to ensure no additional issues arise.
07-07-2010 9:18 AM Yellow cTV: partial service interruption
A number of cTV channels (MSNBC, CNN, PBS, WGN, ABC and CBS) are inaccessible following planned maintenance and hardware upgrades during the day of July 6th. IT Services staff are currently working with the vendor to resolve the issue.
06-22-2010 8:33 AM Green Network Connectivity Problems Resolved

UPDATE: IT Services' Network Engineers report that this morning's network problem was a result of misconfiguration of a switch during installation. IT Services is reviewing processes to ensure this does not happen again. We apologize for the inconvenience and interruption earlier today.

As of approximately 8:25AM Network Connectivity has been restored. IT Services' Network Engineers are continuing to investigate the cause of this morning's problems.

06-22-2010 8:16 AM Yellow Intermittent Network Connectivity Problems

Some parts of the campus network are experiencing intermittent connectivity problems as of approximately 8AM. IT Services Network Engineers are actively investigating these problems. We will post an update when more information is available and provide regular status updates.

Service affected may include: VOIP Phones, Wired & Wireless network connectivity and access to any services which require network connectivity.

Some locations that have reported issues: Booth, Foster, McGiffert & 6054 Drexel.

05-30-2010 10:11 AM Green Wireless Network Maintenance Complete
As of 7:00 a.m on Sunday, May 30, maintenance on the wireless network is complete and the wireless network is available for use.
05-28-2010 2:53 PM Yellow Wireless Network Maintenance on May 30

On the morning of Sunday, May 30 - from 6:30 AM to 7:30 AM - IT Services will perform vendor-recommended maintenance on the wireless gateways to address slow assignment of IP addresses to wireless users. During the maintenance window wireless users may experience disconnects and periods of up to ten minutes when wireless connections are not possible.

We apologize in advance for this disruption, and thank you for your patience while we work to improve our services.

05-11-2010 3:54 PM Green cMail Service Restored

UPDATE as of 4:15PM: Email messages that were delayed may be received out of order, but will have the correct timestamps. Additionally, cMail performance may be impacted while the backlog is cleared.

IT Services systems staff completed a restart the cMail servers at approximately 3:40PM on Tuesday, May 11, which has restored the service to normal operation. Email messages sent while the cMail system was experiencing problems were queued for delivery and delayed for up to 1 hour - these messages will be delivered as the backlog is cleared.

05-11-2010 3:14 PM Yellow cMail Login Issues
At approximately 2:45PM on Tuesday, May 11 cMail began experiencing issues which present to various email clients as timeouts, slowness and/or an inability to login. IT Services systems staff are currently working to resolve the problem.
04-23-2010 1:07 PM Green cMail Operating Normally
After closely monitoring the performance of cMail this morning, we are no longer seeing evidence of performance problems. IT Services staff will continue to monitor this service and remain in contact with the vendor about it. Thank you for your patience with this situation.
04-23-2010 8:50 AM Yellow cMail Slowdown Update
IT Services staff restarted the problematic portion of the cMail system earlier this morning. We are no longer seeing evidence of performance problems, however we continue to monitor this service and remain in contact with the vendor about it. We will be closely monitoring the performance of this service for the rest of the morning and will post an update at approximately 1PM. Thank you for your patience as we work through this situation.
04-22-2010 2:46 PM Yellow cMail Slowdown
At approximately Noon on Thursday, 22 April, one of the mail storage units on cMail, the University's central email system, began experiencing performance issues. As a result, approximately one-fourth of the people using cMail may experience symptoms ranging from slow performance to an inability to access their mail. IT Services systems staff are currently working with the vendor to resolve the problem.
04-21-2010 3:00 PM Green Faulty McAfee Virus Definitions and Windows XP Systems

At approximately 8AM local time on April 21, McAfee released a faulty update to the virus definitions used by Viruscan Enterprise. Windows XP systems running a corporate version of McAfee Viruscan - including the version available from antivirus.uchicago.edu - may experience severe issues.

If your Windows XP system began experiencing unusual problems earlier today, such as blue screen crashes or DCOM errors, followed by shutdown messages, please read on for additional information and troubleshooting steps.


(Read More...)
03-31-2010 10:57 AM Green Intermittent Login Problems
On the morning of Wednesday, 31 March, IT Services began receiving reports of intermittent problems accessing services using CNetID and CNet password. The issue was traced to an unexpectedly high load on the University's central authentication service (LDAP). As of 10:45 AM, the authentication service has been restarted and the problem seems to be resolved; however, IT Services staff continue to monitor the service closely and are investigating the source of the load spike to ensure this does not recur.
03-24-2010 9:17 AM Green Chalk Course Creation Restored
As of 8:30 AM Wednesday, 24 March, new course creation for Chalk is working normally. IT Services are now working on backlogged course creation requests; this process should be caught up by the end of the day. Thank you for your patience while we resolved this issue.
03-23-2010 9:43 AM Yellow Chalk New Courses - Update
IT Services staff continue to work on a technical problem which is preventing us from creating new courses in Chalk. We anticipate that this issue will be resolved by the beginning of the business day on Wednesday, 24 March. Thank you for your patience as we work through this situation.
03-22-2010 9:21 AM Yellow Chalk Issues
On the morning of Monday, 22 March, Chalk's scheduled maintenance window had to be extended; as a result, it was unavailable until 8:30 AM. It is currently back up and available, but the ability to create new courses is still unavailable. All other Chalk features are working normally. IT Services staff are working to restore new course creation as soon as possible.
03-11-2010 1:13 PM Green CNetID Maintenance Claiming Issues Resolved
On Wednesday, 10 March, three separate issues began affecting CNetID claiming and maintenance processes. The cumulative result of these issues could cause silent failures for people who attempted to claim a new CNetID; to change their CNet password; or to change their email forwarding/email aliases. These problems were brought to NSIT's attention on the morning of 11 March. As of 1 PM on 11 March, all three issues have been resolved, and all CNetID claims and maintenance services should now be working normally. We are revisiting our monitoring and testing processes to help prevent these kinds of issues.
03-11-2010 12:23 PM Yellow CNetID Maintenance/Claiming Issues
On the morning of Thursday, 11 March, NSIT began receiving reports of problems claiming CNetIDs and with maintaining information connected to the CNetID (password changes, email forwarding, and email aliases). Users attempting to create a new CNetID, or change their CNet password, email aliases, or email forwarding destination, may receive messages that their claim or change has completed successfully when it has not. Note that actual email forwarding is NOT affected; this problem only affects attempts to change where email is being forwarded to. NSIT systems staff are investigating the problem.
03-05-2010 4:41 PM Green Wireless on South Campus Working
On the afternoons of Wednesday and Thursday, March 3rd and 4th, NSIT received several reports of wireless problems on the south side of the main Quads, around Booth and Judd, and in buildings south of the Midway. This included sudden disconnections from wireless, and/or being unable to connect to wireless at all. On Friday March 5th, we installed a vendor-provided fix targeted at a potential problem connected to the number of concurrent users. As of 4:30 PM Friday, we have received no further reports of issues with wireless, and we believe the problem to be resolved. NSIT network engineers will continue to closely monitor the situation for the next few days to ensure that the fix works as the number of users reaches its normal Monday/Tuesday peak.
03-04-2010 4:44 PM Yellow Wireless Issues on South Campus
On the afternoon of Wednesday, 03 March, NSIT began receiving sporadic reports of wireless issues on the South side of the main quads and South of the Midway. This included sudden disconnections from wireless, and/or being unable to connect to wireless at all. The problem became worse through Thursday 04 March. As of approximately 3:45 PM Thursday, NSIT network engineers have isolated the issue to a problem with one of the wireless authentication servers, and moved a backup server into place. We believe this has remedied the problem people have been seeing, and put us in a stable position until we can put in a permanent fix. If you are still seeing wireless issues, please contact NSIT Support at 773-834-8324.
03-02-2010 9:40 PM Green Services Restored
Overnight between Monday, March 1st, and Tuesday, March 2nd, an intermittent network problem in the NSIT Data Center began causing recurring problems in several NSIT services. As of 9:30 PM, the network in the Data Center has been reconfigured to isolate the network issue, and we believe that we have put in a workaround. NSIT staff will continue to monitor services through tonight and tomorrow to confirm normal operation, and will also work with vendor support to find the root cause of the problem and ensure it does not happen again.
03-02-2010 7:35 PM Yellow xMail restored; update on Intermittent Outages
Overnight between Monday, March 1st, and Tuesday, March 2nd, an intermittent network problem in the NSIT Data Center began causing recurring problems in several NSIT services. As of 7:30 PM, March 2nd, NSIT network engineers have put configuration changes in place in an attempt to remedy these problems; NSIT staff are currently testing applications to confirm that they are operating normally. Services potentially affected include Chalk, xMail, Griffin, the Library online catalog, some Tank filestores, cEikon, eLedgers, and Business Objects, among others.
03-02-2010 4:48 PM Yellow Intermittent Service Outage - Update
Overnight between Monday, March 1st, and Tuesday, March 2nd, an intermittent network problem in the NSIT Data Center began causing recurring problems in several NSIT services. As of 4:30 PM, March 2nd, NSIT network engineers are planning to institute a set of configuration changes to the relevant network segment in an attempt to mitigate or resolve the problem. These changes will be made through Tuesday evening, and may result in brief outages, hopefully followed by improved service. Affected services include Chalk, xMail, Griffin, the Library online catalog, some Tank filestores, cEikon, eLedgers, and Business Objects, among others.
03-02-2010 9:44 AM Yellow Intermittent Service Outages
An intermittent network problem in the NSIT Data Center is causing recurring problems for several NSIT services. The network traffic is occasionally spiking; whenever these spikes happen, services on the affected network segment may see poor performance or may become completely unavailable. Spikes typically last 15 to 45 minutes. Affected services include the Library online catalog, Griffin, some Tank filestores, cEikon, eLedgers, and Business Objects, among others. NSIT network engineers are working with vendor techs to isolate and repair the problem.
03-01-2010 9:53 AM Green Services restored
At approximately 7:15 AM Monday, 1 March, a hardware failure on our central data storage system caused several services to become unavailable or to experience slow performance. Services affected included My.uchicago.edu, the proxy system used to access Library resources, lists.uchicago.edu, im.uchicago.edu, and bulkmail, among others. As of 9:45 AM, all services have been restored. NSIT systems staff are working with vendor support to determine why redundant hardware did not properly take over at the time of the initial failure.
03-01-2010 9:23 AM Red My.uchicago, other service outage
At approximately 7:15 AM Monday, 1 March, a hardware failure on our central data storage system caused several services to become unavailable or to experience slow performance. Services affected included My.uchicago.edu, the proxy system used to access Library resources, lists.uchicago.edu, im.uchicago.edu, and bulkmail, among others. NSIT systems staff are working to restore systems access; as of 9:15 AM, My.uchicago and many other services have been recovered, and it is anticipated that all services will be restored by no later than 10 AM.
02-15-2010 2:22 PM Green Resolved: cIM Outage.
cIM is once again available. Although preliminary diagnosis indicates a hardware failure may be at fault, NSIT staff continue to investigate the core cause of the outage.
02-15-2010 12:41 PM Yellow cIM Outage
cIM, the University's central instant messaging service, is currently unavailable. NSIT systems staff are working to resolve the problem.
02-15-2010 10:20 AM Green DNS Service Update
We have completed the diagnostic tests on the University's central domain name service (DNS), developed a set of preventative measures to minimize the chance for future outages of the type encountered last week, and are applying a series of updates to the servers that provide DNS to campus. We anticipate finishing the update process on Friday, 19 February. Because of the nature of the updates that are being applied to the servers, we have temporarily suspended additions and changes to DNS. Therefore, any changes to DNS such as adding new hosts, requesting static IP addresses, or changing hostnames are being queued until we will resume processing DNS requests on Friday, 26 February.
02-12-2010 12:55 PM Yellow DNS Service Testing
Beginning the afternoon of Friday, 12 February, NSIT will be performing diagnostic tests on the University's central domain name service (DNS). The diagnostics should be largely invisible; however, due to the nature of the tests, you may see occasional small delays when accessing services on or from the campus network. The tests will be run on Friday afternoon and during the business day on Monday and Tuesday (we need normal daytime traffic to validate the testing). We apologize for any inconvenience this may cause; we hope, however, that information from these tests will allow us to isolate the cause of this past Wednesday's DNS outage, and install a definitive fix to prevent it from happening again.
02-10-2010 8:17 PM Green DNS Service Outage Resolved
At approximately 1:30 PM Wednesday, February 10th, the University's primary domain name service (DNS) began experiencing an extremely high load. This caused general problems for on-campus users trying to connect to web-pages, email, and other services, both on- and off-campus; and problems for off-campus users trying to connect to on-campus services. The problem was tracked to corruption in the underlying DNS database. The database was restored from backup, and as of 7:30 PM, the problem has been resolved. Some email sent to uchicago.edu addresses during the outage may have bounced back to the sender; these messages will need to be re-sent. Apart from that, all services should now be operating normally.
02-10-2010 2:24 PM Red DNS Service Outage (6 PM Update)
At approximately 1:30 PM Wednesday, February 10th, the University's primary domain name service (DNS) began experiencing an extremely high load. As of 6 PM, the issue is still ongoing; NSIT staff are still attempting to isolate the problem, but in the meantime, we are also bringing up additional back-end servers in an attempt to further distribute the load. Since DNS is used to help local computers find other machines on the Internet, until this problem is resolved, campus users will have difficulty connecting to web pages and other services, both on-campus and off-campus, unless their local machine has already cached information about the remote site. Additionally, off-campus users may have similar problems accessing any on-campus services. This may result in email to uchicago.edu addresses being bounced back to the sender.
02-10-2010 10:11 AM Green Blackberry Email Service Interruption
After routine maintenance during the morning of 2/10, a synchronization issue arose between xMail and the server that distributes email to Blackberries. This issue was quickly discovered and fixed as of 9:30am. No messages were lost, though some correspondence may have been delayed. We are examining our maintenance processes to ensure this does not happen again.
02-05-2010 8:53 AM Green cIM Instant Messaging Service Interruption
Earlier this morning the cIM instant messaging service became unavailable. Service was restored at 8:10AM. NSIT administrators are investigating this outage to prevent recurrence.
02-03-2010 10:57 PM Green Network Outage Resolved
Between 8 and 8:30 PM February 3rd, a denial-of-service attack effectively cut the campus network off from the Internet at large. NSIT staff successfully identified the (off-campus) source of the attack and blocked it; as of 11 PM, the campus network appears to have fully recovered, and normal access is restored.
02-01-2010 5:49 PM Yellow General form mailing script service update
Disruption to the general form mailing script service continues. On the afternoon of January 27, an apparent attack on the general form mailing script impacted the main University web server. To restore the performance of the server and eliminate impact on University web sites, the script was removed and a new script designed to replace it. Service has been restored to those sites known to be using the old script, and efforts to identify others continue. NSIT is working to resolve this problem for all affected sites as soon as possible. Please refer to the Campus Notices page for continuous updates as information becomes available. If you are the owner of a web site including a form powered by the University's general form mailing service, please contact Web Services at nsit-webserv@lists.uchicago.edu with the URL of your affected web form so that we may assist you.
01-28-2010 4:27 PM Yellow cMail Slowdown
On the afternoon of Thursday, January 28th, two separate events -- an unusually large spike of incoming email from several sources, and, later, an incorrectly triggered backup process -- led to extremely slow performance on cMail, the University's primary email system. As of 2:30 PM, both events were brought under control, and cMail performance was restored to normal. NSIT staff are taking steps to ensure that the incorrect backup process does not recur, and also acting to upgrade the cMail system to better handle email spikes of this magnitude.
01-28-2010 11:09 AM Yellow Spike in network traffic impacts DHCP service

At about 8:30AM on Thursday, 28 January, an increase in traffic to or from campus routers was noted. As a result, some areas began experiencing issues with network connectivity related to an inability to communicate with the DHCP service. We've received reports from clients in the Gordon Center, 6045 S. Kenwood, 6030 S. Ellis, Searle and Edelstone with connectivity issues.

As of approximately 10:15 AM the network load has cleared and DHCP clients are no longer having connection issues. If you continue to experience connectivity issues, please restart your computer to obtain a current IP address from the DHCP service. NSIT network engineers continue to monitor the situation to prevent this problem from recurring.

01-27-2010 5:40 PM Yellow General form mailing script service interrupted again
On the afternoon of January 27, NSIT noticed a sharp increase in web traffic related again to an apparent attack on the general email script used on many websites across the University. The problem began at approximately 4:30 pm and resulted in slowdown of websites hosted on the main University web server. NSIT staff is working to resolve the issue and has again blocked the general email script from use as a security measure. More information and a confirmation of the resolution will be posted shortly. We apologize for the inconvenience.
01-26-2010 6:06 PM Green General form mailing script service reinstated
At approximately 5:00 PM on January 26, NSIT reinstated the general email script service. NSIT programming staff added a CAPTCHA requirement to all forms that are processed by the general email script, thus changing the user-experience for the script. All users submitting a form that is processed by the general email script will be required to enter one additional form field of letters and numbers as represented within an image file displayed to the user. This is a standard tool used to combat spam and other abuses of publicly-facing web forms. If you experience any difficulties in submitting forms with the CAPTCHA requirement or notice any unusual behavior of certain web forms, please report these details along with your name and the name and version of the browser you were using to weberror@lists.uchicago.edu. Thank you for your patience while the issue was being resolved, and we apologize for the inconvenience.
01-26-2010 1:22 PM Yellow General form mailing script service interrupted
On the afternoon of January 26, NSIT noticed a sharp increase in spam coming through the general email script used on many websites across the University. The problem began at 11:30 am and resulted in a slowdown of email services. The issue has been identified as a vulnerability in the general email script. NSIT staff is working to resolve the issue and has blocked the general email script from use as a security measure until it can be secured. More information and a confirmation of the resolution will be posted shortly. We apologize for the inconvenience.
01-14-2010 3:02 PM Green Internet Service Interruption Resolved
On the morning of Thursday, January 14th, a router problem with Cogent (one of the University's Internet service providers) caused several Internet sites (including Google) to become inaccessible from the campus network. The problem lasted from approximately 11 AM until 11:45 AM. As of 2:30 PM, Cogent has confirmed that the problem is fixed, and we should have no further issues.
01-14-2010 12:08 PM Yellow Internet Service Interruption
At approximately 11 AM Thursday, 14 January, one of the University's commodity links to the outside Internet stopped working correctly. As a result, many off-campus sites (including Google) became inaccessible from the campus network, and others responded slowly or inconsistently. Cogent (the provider in question) identified the problem as an issue with one of their Chicago-area routers (affecting many of their Chicago-area customers). Cogent put in a fix at approximately 11:45 AM; the situation now appears to be stable, but we are still waiting for confirmation from Cogent that the situation is resolved.
12-21-2009 9:33 AM Green All Systems Normal
Systems are currently functioning normally.
12-18-2009 5:21 PM Yellow Data Center Network Maintenance - Dec. 19-20
In the early morning hours of December 20th, NSIT will be performing a series of network configuration changes to its central data center to improve performance and stability. Enacting these changes will cause some disruption for many NSIT services. Specifically: xMail, NSIT's Exchange-based email service, will be unavailable from 11:45 PM CT December 19th, until 12:30 AM CT December 20th (a total downtime of 45 minutes). Additionally, most other NSIT services will experience brief disruptions (between 3 seconds and 90 seconds) during the period between 12:01 AM and 2 AM CT December 20th. Some services may experience multiple disruptions during that timeframe. We apologize in advance for this disruption, and thank you for your patience while we work to improve our services.
12-18-2009 9:27 AM Green Partial cMail Outage Resolved
The condition which precipitated yesterday's partial cMail outage has been identified and a temporary fix has been applied. NSIT systems staff continue to monitor the cMail system and a permanent fix will be applied during a normal maintenance period.
12-17-2009 12:19 PM Yellow Partial cMail Outage
At approximately Noon on Thursday, 17 December, one of the mail storage units on cMail, the University's central email system, began experiencing connectivity issues. As a result, approximately one-fourth of the people using cMail temporarily lost access to their mail. NSIT systems staff are currently working with the vendor to resolve the problem.
12-15-2009 10:51 PM Yellow High traffic to web applications resolved
Issues related to high traffic to the main University web server have been resolved. Follow the link for more details.
(Read More...)
12-15-2009 4:41 PM Red High traffic affecting web applications
At approximately 3 pm on Tuesday, 12/16/09, performance of web applications hosted on the University's main web servers, such as cMore and the College Admissions website, became slow due to unexpectedly high traffic. NSIT is working to investigate and resolve the problem. We currently have no estimate for when full performance will be restored. Please refer to NSIT traffic lights http://www.uchicago.edu/notices for continuous update as information becomes available. To access the NSIT traffic light page, visit the University of Chicago home page (http://www.uchicago.edu) and click on "Campus Notices" at the bottom of the page.
10-30-2009 9:38 AM Green Find IT Restored
At approximately 9 a.m. CDT on 30 October 2009 Find IT has been restored to normal functionality.
10-29-2009 4:40 PM Yellow Find IT Unavailable
Find IT, NSIT's service catalog, is currently unavailable. We are in communication with the vendor to resolve this problem.
10-27-2009 10:19 PM Green VPN Logins Restored
On the afternoon of Tuesday, October 27th, NSIT began receiving reports of isolated problems logging in to the University VPN. The problem was tracked to a missing attribute on the users' CNetIDs. Further investigation revealed that approximately 10% of our users were incorrectly missing this attribute. The root cause of the problem was corrupted data in the database managing the VPN access group. As of 6:15 PM, this corruption has been cleaned out, correct information restored, and all eligible users can once again access the VPN.
10-27-2009 5:27 PM Yellow Email Issues Resolved; VPN issue still ongoing
Email being sent to @uchicago addresses is rotated through one of three mail servers for delivery. At approximately 2 PM Tuesday, 27 October, a cache problem on one of these servers caused incoming messages routed through that server to be returned to their senders. As of 5:15 PM, the cache problem has been cleared, and normal email delivery has been restored. This problem was unrelated to problems some users are having logging on to the VPN; NSIT staff are still working on that issue.
10-27-2009 4:59 PM Yellow Isolated VPN Login Problems
NSIT has received complaints that some users are unable to log in to the University VPN service. The issue has been traced to a missing attribute on the users' CNetIDs; further investigation suggests that this attribute is missing for around 10% of our users. NSIT staff are currently working to restore this attribute and normal VPN access. This problem is unrelated to ongoing problems with email delivery to @uchicago.edu addresses.
10-27-2009 4:53 PM Yellow Sporadic Bounces to @uchicago.edu addresses
NSIT has received reports of sporadic problems with email delivery; some messages sent to @uchicago.edu addresses are being returned to senders. The problem has been traced to one of the three incoming email servers we have in rotation, and NSIT staff is currently working to resolve the issue.
10-08-2009 2:50 PM Green Resolved: Slow Performance on cMail
NSIT Systems Engineers identified the cause for the slowness impacting a portion of the cMail system which began earlier this afternoon. An automated backup system erroneously began backing-up the storage system for one of the backend mail systems, which degraded the performance for users on that system. NSIT Systems Engineers cancelled the erroneous backup job at approximately 1:45PM. The impacted mail system has cleared its backlog and resumed normal operation.
10-08-2009 2:03 PM Yellow Updated: Slow Performance on cMail
NSIT Systems Engineers have identified the cause for the slowness impacting a portion of the cMail system which began earlier this afternoon. An automated backup system erroneously began backing-up the storage system for one of the backend mail systems, which degraded the performance for users on that system. NSIT Systems Engineers cancelled the erroneous backup job at approximately 1:45PM. Individuals may experience degraded performance as the impacted mail system returns to normal operation.
10-08-2009 1:35 PM Yellow Slow Performance on cMail
NSIT is aware of recent performance issues on cMail, our central standard email system. This tends to take the form of long delays or hangs when logging in or accessing email for some, but not all users. NSIT Systems Engineers are investigating the issue and we will post more information as it becomes available.
09-13-2009 11:20 PM Green Chalk Restored
Chalk, the University's central learning management system, experienced a filesystem failure during standard maintenance on the morning of Sunday, 13 September. As a result, Chalk was unavailable for most of the day Sunday while NSIT staff attempted to recover the drive; in the end, they were forced to reformat and restore from the Friday night backup. As of 11:15 PM, Chalk is once again fully functional. Files uploaded on Saturday will need to be re-uploaded; other changes (e.g. forum postings or tests/surveys) should be fine, as they would not have been on the affected drive.
09-13-2009 6:16 PM Red Update (6:15 PM): Chalk Outage
Chalk, the University's central learning management system, experienced a filesystem failure during standard maintenance on the morning of Sunday, 13 September. As of 6:15 PM, NSIT staff are restoring Chalk from last night's backup. Chalk is expected to be back up before Monday morning.
09-13-2009 3:31 PM Red Update: Chalk Outage
Chalk, the University's central learning management system, experienced a filesystem failure during standard maintenance on the morning of Sunday, 13 September. As of 3:30 PM, Chalk is still unavailable; NSIT staff are working with vendor support to recover the system.
09-13-2009 11:43 AM Red Chalk Outage
As of 11:30 AM 13 September, Chalk, the University's central learning management system, is unavailable due to problems with its underlying filesystem. NSIT staff are working to resolve the problem.
09-04-2009 2:15 PM Green cTV Maintenance Complete
As of 2:00 p.m. on Friday, September 4, 2009, the scheduled cTV maintenance is complete and cTV is available for viewing.
09-03-2009 11:57 PM Yellow cTV Maintenance
cTV will be unavailable on Friday, September 4, 2009 due to vendor-required maintenance.
08-27-2009 2:34 PM Green Find IT Restored
At approximately 1 p.m. CDT on 27 August 2009, Find IT has been restored to normal functionality.
08-27-2009 11:19 AM Yellow Find IT Unavailable
Find IT, NSIT's service catalog, is currently unavailable. We are in communication with the vendor to resolve this problem.
08-25-2009 4:06 PM Green Systems Restored
At approximately 3:00 am CDT on 25 August 2009, there was an isolated storage system failure that impacted a number of smaller servers including formulator.uchicago.edu, wiki.projectbamboo.org, and the organization and help request forms used by Chalk. The issue was traced to a configuration problem, which has now been remedied; as of 4 PM, all systems should now be running normally.
08-25-2009 12:24 PM Yellow FindIT restored; other systems still down
As of approximately 12 noon, 25 August, FindIT has been restored to normal functionality. However, the original incident, involving an isolated storage system failure that impacted a number of smaller servers including formulator.uchicago.edu, wiki.projectbamboo.org, and the organization and help request forms used by Chalk, is still being worked on. This particular outage is limited only to these minor systems; Chalk and other major systems are unaffected. NSIT staff are working to restore these services as soon as possible.
08-25-2009 9:42 AM Yellow Storage Failure issues on some systems; FindIT also down
At approximately 3:00 am CDT on 25 August 2009, there was an isolated storage system failure that impacted a number of smaller servers including formulator.uchicago.edu, wiki.projectbamboo.org, and the organization and help request forms used by Chalk. This particular outage is limited only to these minor systems; Chalk and other major systems are unaffected. NSIT staff are working to restore these services as soon as possible. In an unrelated incident, FindIT, NSIT's service catalog, is down. We are in communication with the vendor to resolve this problem.
08-25-2009 9:04 AM Yellow Storage Failure Problems on several systems
At approximately 3:00 am CDT on 25 August 2009, there was an isolated storage system failure that impacted a number of smaller servers including formulator.uchicago.edu, wiki.projectbamboo.org, and the organization and help request forms used by Chalk. This particular outage is limited only to these minor systems; Chalk and other major systems are unaffected. NSIT staff are working to restore these services as soon as possible.
08-14-2009 8:52 AM Green Resolved: xMail Service Issues
NSIT staff isolated the cause of the instability in the xMail service at 6:30pm on Thursday, August 13th, and took action to repair the xMail service. As of this notice, the xMail service is once again fully operational and all individuals should have access. We remain in contact with the vendor and are closely monitoring the servers to ensure that this problem does not recur.
08-13-2009 4:32 PM Yellow xMail Service Issues - Update
Beginning late in the afternoon of Wednesday, 12 August, two of the front-end servers to xMail, NSIT's central Exchange service, began experiencing problems, which have been growing progressively worse. This issue causes sporadic disconnects or login problems for people trying to access xMail using IMAP (e.g. with Thunderbird) or over the web. As of 4:30 PM Thursday, 13 August, the problem has expanded to include issues sending mail with Outlook -- Outlook clients can see existing mailboxes and incoming email, but cannot consistently send mail. NSIT is continuing to work with the vendor to resolve this issue.
08-13-2009 2:50 PM Yellow Service Issues on xMail
Beginning late in the afternoon of Wednesday, 12 August, two of the front-end servers to xMail, NSIT's central Exchange service, began experiencing problems, which have been growing progressively worse. This issue causes sporadic disconnects or login problems for people trying to access xMail using IMAP (e.g. with Thunderbird) or over the web. Users with a standard Outlook configuration may see occasional slow performance. NSIT is currently working with the vendor to resolve the problem.
08-10-2009 8:59 AM Green Early Monday cMail Outage

A significant portion of the cMail system lost the ability to authenticate clients from approximately 1:00AM until 7:00AM, early Monday morning. The affected components of the service were restarted at 6:15AM and access was restored to all clients shortly thereafter.

NSIT Staff are consulting with the mail system vendor to determine why this interruption in service occurred.

08-05-2009 7:09 PM Green Gargoyle Student System - Service restored
Service to the Gargoyle Student System was restored at approximately 6:30pm. The application is now available for use. NSIT will be conducting a root cause analysis to identify and understand the cause of this outage.
08-05-2009 4:08 PM Red Gargoyle Student System Disruption
The Gargoyle system is experiencing a problem, we are aware of it and system administrators are working to find what is causing this and to resolve it as quickly as possible. Please logoff of the Gargoyle system and remain off the system until further notice. We will send out another email to all users when it is OK to use Gargoyle again. We thank you in advance for your patience and understanding at this time.
07-29-2009 1:43 PM Green Resolved cMail Outage
While conducting an upgrade on the storage servers that support cMail, an unforeseen configuration conflict caused part of the cMail service to fail. With on-site vendor assistance, the configuration conflict was resolved and the cMail service was restarted successfully shortly after 1PM. No email was lost as a result of this outage.
07-29-2009 12:51 PM Red cMail Outage
At approximately 12:30 PM Wednesday, 29 July, two of the mail storage units on cMail, the University's central email system, stopped responding. As a result, approximately half of the people using cMail are currently unable to check their mail. NSIT systems staff are currently working with the vendor (who is on-site) to resolve the problem.
07-21-2009 12:30 PM Green CNN Restored on cTV
The faulty hardware component has been replaced and CNN is available. The cTV service is now functioning normally.
07-20-2009 10:32 AM Yellow cTV CNN Problem
The component of the cTV service, which provides CNN, is currently experiencing a hardware problem. As a result, Channel 100 (CNN) is not available at this time. NSIT is working with the vendor to obtain replacement hardware, which is currently expected to arrive tomorrow morning. CNN should be available again before NOON tomorrow (7/21).
07-13-2009 10:12 AM Green Resolved: Bouncing Email on lists
Reminder: Any email sent to a list on lists.uchicago.edu during the ~35 minutes of the outage on Friday should have been returned to the individual who sent it; they will need to re-send the message to make sure it goes through.
07-10-2009 12:20 PM Yellow Bouncing Email on lists
On 10 July, at approximately 11:40 AM, a configuration error on lists.uchicago.edu began causing email sent to mailing lists to bounce. The problem was discovered and fixed at approximately 12:15 PM. Any email sent to a list on lists.uchicago.edu during the ~35 minutes of the outage should have been returned to the individual who sent it; they will need to re-send the message to make sure it goes through.
07-01-2009 12:10 PM Green Resolved: Authentication Problems
Beginning around 8:45am on July 1, many people on campus became unable to authenticate to the University wireless network. Others were also unable to access webspace.uchicago.edu. The problem was traced to an error that occurred when propagating eligibility information to our authentication systems. As of 11:45am, NSIT has restored all access by repopulating the eligibility fields.
07-01-2009 10:50 AM Yellow Wireless Authentication Problem -- Update
On the morning of July 1, a large number of people on campus became unable to authenticate to the University wireless network. The problem has been traced to an error which occurred when propagating wireless eligibility to our authentication system. Systems staff are currently repopulating the eligibility fields; people will be able to log in again as the correct data propagates. Unfortunately, the process is taking longer than we anticipated; we currently project that the problem will be resolved by 1 PM. The error did not affect any service other than wireless access.
07-01-2009 9:57 AM Yellow Wireless Authentication Problems
A large number of people on campus are currently unable to authenticate to the University wireless network. The root of the problem has been traced to an error which occurred when propagating wireless eligibility to our authentication system. Systems staff are currently repopulating the eligibility fields; people will be able to log in again as the correct data propagates. The problem should be entirely resolved by 10:30 AM. The error did not affect any service other than wireless access.
06-15-2009 10:48 AM Green NSIT service outage on 6/11/2009 that impacted campus websites
This message is a follow-up to the outage on Thursday, June 11 that severed access to over 1,000 campus websites from about noon to 5:30 pm. We want to share with you, what happened, the solution process, and plans to ensure that this type of incident does not re-occur.
(Read More...)
06-11-2009 5:44 PM Green Resolved: Web hosting environment outage
As of 5:35pm on 6/11, NSIT personnel have implemented a solution to restore the web hosting environment's full functionality. All sites are once again accessible.
06-11-2009 2:33 PM Red Updated: Web hosting environment outage
As of noon on June 11th, NSIT began experiencing DNS-related problems with one of the web hosting environments. Many websites, primarily those hosted on webspace.uchicago.edu, are presently unavailable. The source of the problem has been identified and NSIT personnel continue to work in conjunction with an outside vendor. At this time, we do not anticipate a resolution before the close of business today, 6/11.
06-11-2009 1:03 PM Red Web hosting environment outage
NSIT is presently experiencing problems with one of the web hosting environments. Many websites, primarily those hosted on webspace.uchicago.edu, are presently unavailable. The source of the problem has been identified and NSIT personnel expect to resolve the problem before 2pm.
05-14-2009 1:40 PM Green USITE, cMail Performance Issues Resolved
After consultation and service calls from both hardware and software vendors, the cluster of servers supporting the USITE workstations is again operating at 100% capacity. Service was not interrupted during the course of this incident. Additionally, the vendor-installed patch on the morning of 13 May appears to have resolved performance issues for cMail users; if you are still seeing hangs or slow performance using cMail, please contact the NSIT Support Line at support@uchicago.edu or 834-TECH.
05-11-2009 3:52 PM Yellow Slow performance on USITE Workstations
We are experiencing a hardware problem with one of the four servers which provide USITE users with their Windows environment. We are investigating the problem to determine the cause and severity. In the mean time, the faulty system has been removed from the cluster. All system users may notice slightly slower response time from their workstation during peak utilization hours, when load on the remaining members of the cluster will increase.
05-07-2009 4:40 PM Yellow Slow Performance on cMail
NSIT is aware of recent performance issues on cMail, our central standard email system. This tends to take the form of long delays or hangs when logging in or accessing email. We have spoken with the vendor, and they are providing us with a patch which they say will address the issue; they will be installing this patch during our normal early morning maintenance period on Wednesday, 13 May. In the meantime, we have noted that the problem seems to be much worse for people using webmail.uchicago.edu than for people using desktop clients (such as Thunderbird), so you may see improved performance if you try a different client; instructions for configuring desktop clients can be found at http://nsit.uchicago.edu/docs/email/configuration/
05-04-2009 9:45 AM Green Harper back up; home still read-only
At approximately 2:45 PM Thursday, 30 April, a disk failure began causing problems for users of harper, NSIT's central interactive Unix system. NSIT systems staff were forced to do an emergency shutdown to deal with the failure at approximately 3:30 PM, which also interrupted service for home.uchicago.edu, NSIT's central personal web page service. As of 9 AM Monday, 4 May, harper is back up and running normally with files restored from the backup done on the night of 29 April. Web pages on home.uchicago.edu can be read normally over the web, but they cannot be updated; we are working to make these two services more independent of each other before fully restoring write access to home pages. We will contact home page owners shortly with more information.
05-01-2009 5:31 PM Yellow Harper partially restored; home.uchicago.edu still read-only
At approximately 2:45 PM Thursday, 30 April, a disk failure began causing problems for users of harper, NSIT's central interactive Unix system. NSIT systems staff were forced to do an emergency shutdown to deal with the failure at approximately 3:30 PM. As of 5 PM Friday 01 May, harper's user files have been restored to their status as of Wednesday night, 29 April, and logins to harper have been turned back on. Work to stablize the system is continuing, however, and the machine will be periodically unavailable over the weekend due to reboots. Additionally, although home.uchicago.edu is serving files out to the web, it is not possible to update or change pages on home (via public_html) at this time. More information will be posted here as it becomes available.
05-01-2009 9:27 AM Yellow Harper unavailable; home.uchicago.edu disruptions possible
At approximately 2:45 PM Thursday, 30 April, a disk failure began causing problems for users of harper, NSIT's central interactive Unix system. NSIT systems staff were forced to do an emergency shutdown to deal with the failure at approximately 3:30 PM, and the system has been unavailable since that time. As of 9 AM Friday, 01 May, NSIT systems staff have succeeded in recovering data from the lost disk, but are still working to get the disk array back online; this may lead to brief disruptions in the availability of web pages on home.uchicago.edu. We hope to have harper restored by noon. People who access their email using harper should be able to access their new email over the web (at https://webmail.uchicago.edu/ ) in the interim.
04-30-2009 5:29 PM Yellow home restored; harper still unavailable
At approximately 2:45 PM Thursday, 30 April, a disk failure began causing problems for users of harper, NSIT's central interactive Unix system. NSIT systems staff were forced to do an emergency shutdown to deal with the failure at approximately 3:30 PM; this shutdown also had the effect of shutting down home.uchicago.edu, the central home page service for the University. As of 5:30 PM, home.uchicago.edu is back up in a read-only state (i.e. it is not be possible to update pages, but they are viewable over the web). Systems staff are continuing to work on harper, but we do not expect it to be available again before tomorrow. People who use harper for email should be able to access their new mail over the web (at https://webmail.uchicago.edu/ ) until harper is restored.
04-30-2009 4:50 PM Yellow home, harper disk failure
At approximately 2:45 PM Thursday, 30 April, a disk failure began causing problems for users of harper, NSIT's central interactive Unix system. NSIT systems staff were forced to do an emergency shutdown to deal with the failure at approximately 3:30 PM; this shutdown also had the effect of shutting down home.uchicago.edu, the central home page service for the University. Work is currently under way to restore these systems. We expect to have home.uchicago.edu back up later this evening in a read-only state (i.e. it will not be possible to update pages, but they will be viewable over the web). We do not expect harper to be available again before tomorrow. People who use harper for email should be able to access their new mail over the web (at https://webmail.uchicago.edu/ ) until harper is restored.
04-30-2009 12:22 PM Green cMore Restored
Over the night of 29 April 2009, NSIT began receiving reports of login problems on the cMore application. As of 12:15 PM 30 April, a workaround has been put in place and normal logins have been restored. The incident has been tentatively traced to a database access problem, but NSIT staff are continuing to work on a permanent resolution to the underlying issue.
04-30-2009 10:41 AM Red cMore Unavailable
Over the night of 29 April, NSIT became aware of problems with cMore logins. As of 9 AM Thursday, April 30th, all cMore users were experiencing hangs on login. NSIT has disabled logins on cMore while we investigate this issue. More updates will be posted as they become available.
04-20-2009 7:08 AM Green All services restored
At approximately 6 AM Monday, 20 April, a number of machines in NSIT's "virtual" machine service stopped responding. As a result, services running in part or in whole on those machines may be running slowly or may not be available at all. Affected services include Griffin, Gargoyle and other student systems, and the tank1 central Windows file share, among others. As of 7 AM, all the systems in question have been restarted, and all services appear to be operating normally. NSIT systems staff will continue to investigate the source of the problem to ensure it does not happen again.
04-20-2009 6:36 AM Red Service issue for Griffin, Student Systems, tank1
At approximately 6 AM Monday, 20 April, a number of machines in NSIT's "virtual" machine service stopped responding. As a result, services running in part or in whole on those machines may be running slowly or may not be available at all. Affected services include Griffin, Gargoyle and other student systems, and the tank1 central Windows file share, among others. As of 6:30 AM, NSIT systems staff have restarted the systems in question and are monitoring to ensure that systems come back online.
04-02-2009 12:29 PM Green xMail Service Restored
On the morning of Thursday, April 2nd, 2009, a security filter caused a number of xMail users to have trouble accessing their email using Outlook. The problem filter was identified and corrected shortly after noon. NSIT systems staff are investigating the origin of the filter issue and will insert processes to insure this does not happen again.
04-02-2009 11:21 AM Yellow xMail Login Problems
On the morning of Thursday, April 2nd, some users began reporting trouble logging in to xMail using Outlook. Some of these users are able to connect after repeated tries, others are unable to log in at all. We have also received reports of timeouts during continuing sessions. NSIT systems staff is aware of the issue and working to resolve it. As a workaround, we recommend logging in via the xmail.uchicago.edu web interface.
03-24-2009 1:30 PM Green xMail Performance Issues Resolved
NSIT Staff working in conjunction with an external vendor found and replaced a piece of problematic hardware. Replacement of the problematic hardware resolved the performance issues with the xMail service as of approximately 1:15PM.
03-24-2009 9:22 AM Yellow xMail Slowness

NSIT has observed an increasing slowness with the xMail service. We will be taking pre-emptive action at approximately 9:20AM to ameliorate this slowness with the possibility of a potential unscheduled restart.

10:25AM UPDATE: Actions taken by NSIT Engineers have improved xMail performance, but we continue to monitor the situation and may take further action if the slowness redevelops.

02-06-2009 3:15 PM Green Email problems resolved
At approximately 2:30 PM Friday 06 February, one of the email stores on cMail began having severe performance problems. Anyone with email stored on that part of the system (approximately a quarter of the cMail users) was unable to log into or access their email. The issue was resolved at approximately 2:50 PM by a restart of the mail store in question. No email was lost, and as of 3 PM, normal service has been restored. NSIT systems staff are working with the vendor to identify the root cause of the problem and ensure it does not recur.
02-06-2009 2:47 PM Yellow Problems accessing email
One of the mail stores on cMail, NSIT's primary email service, is experiencing problems. Any users who have mail on that store (approximately a quarter of the user base) may have problems logging into or accessing email, or may see severe performance problems. NSIT systems staff are working to identify and resolve the issue.
12-17-2008 9:33 PM Green Voicemail System normal
The university voicemail system was returned to full functionality at approximately 7:15PM this evening.
12-17-2008 4:47 PM Red Voicemail system announces mailbox is full
At about 4:30pm today, the university's voicemail system began reporting users' mailboxes were full and no message could be left by the caller. The matter has been escalated to technical staff who are currently investigating the matter. Updates and resolution will be posted as they become available this evening.

As of 6:30pm, the problem is still under active investigation. Further updates shall follow.

11-20-2008 9:08 AM Green Systems normal
We are not aware of any system problems at this time. If you are still receiving "Relaying Denied" errors when sending email from off-campus, you should update your email configuration; see http://support.uchicago.edu/authsmtp
11-18-2008 5:18 PM Yellow "Relaying Denied" errors sending email from off-campus
On the morning of Wednesday, November 19th, NSIT will block off-campus access to the central outbound email server, smtp.uchicago.edu. After that time, anyone who is still using an email client configured to use that server will receive "Relaying Denied" errors when trying to send email from off-campus. The fix is to update the email settings to use a new, authenticated outbound mail server, authsmtp.uchicago.edu. For instructions, see http://support.uchicago.edu/authsmtp
11-10-2008 12:15 PM Green Voicemail system normal
The voicemail integration has been resolved; users may again directly access their voicemail using the desk phone "Mail" button.

Posted 12:15PM, Monday, November 10, 2008

11-10-2008 11:04 AM Yellow Voicemail access
At about 10:30am, the load on the campus voicemail system spiked and has temporarily altered the way campus users who are accustomed to pressing the "Mail" button from their desk phone may access voicemail. Campus callers are now presented with the greeting customarily given to off-campus callers. To circumvent this temporary problem, users may press the * (star) key and re-enter their 5-digit mailbox number (generally the same as the phone number).

Posted 11:00AM, Monday, November 10, 2008

10-27-2008 12:13 PM Green Systems Normal
As of noon, 27 October, there are no signs that the Windows security problem Microsoft revealed last Thursday is being exploited on campus. However, we still encourage anyone running any version of Windows invoke "Windows Update" (or "Microsoft Update") and make sure all critical security patches are applied. For more information on this security issue, see http://www.microsoft.com/technet/security/bulletin/MS08-067.mspx
10-23-2008 7:08 PM Red Critical Microsoft Security Patch
On Thursday 23 October, Microsoft issued an emergency security bulletin regarding a vulnerability in Windows XP, Windows Vista, and Windows Server products which could allow a remote user to take over local machines and/or use those machines to propagate a worm or virus. Given the extreme nature of the vulnerability and Microsoft's own unusually strong reaction to it, NSIT believes that all Windows machines at the University should be patched against this vulnerability as quickly as possible. We are currently working to update on-campus desktops supported by NSIT. If you use a Windows laptop or have a Windows computer at home, we urge you to update your machine immediately, by going under the Windows or Start menu to "Windows Update" or "Microsoft Update". Look for the critical security update for KB958644. The original Microsoft security bulletin, and instructions for manually patching your machine, can be found at: http://www.microsoft.com/technet/security/bulletin/MS08-067.mspx
10-23-2008 4:49 PM Green Emergency update complete
On the afternoon of Thursday, 23 October, Microsoft released a high-priority alert regarding a previously-undisclosed, severe security hole in the Windows Server operating system. In response, NSIT began an emergency update to all its Windows-based services (including xMail, and tank file shares, among others) at approximately 2:30 PM. The updates may have caused slow performance for some users. All services were updated by approximately 4:30 PM, and are now operating normally. For more information about the security issues resolved by this update, see http://www.microsoft.com/technet/security/bulletin/MS08-067.mspx
10-23-2008 2:59 PM Yellow Emergency updates for xMail, tank fileshares, other services
On Thursday, October 23rd, Microsoft released an alert regarding a severe security hole in current versions of Windows Server, which is the operating system behind xMail, the tank file shares, and several other NSIT services. In response to this alert, NSIT is currently performing an emergency update to apply the newly-released Microsoft patch for this security problem. Although it will not be necessary to disrupt any services while the emergency update is taking place, some users may see slow performance on tank fileshares, xMail, or other services. The emergency update should be complete, and all services restored to normal, by 4 PM.
10-23-2008 2:58 PM Green Off-campus access issues resolved
Early in the morning of 23 October, a routine update to the network name servers unexpectedly caused a small number of external internet server providers (ISPs) to pick up bad information about the uchicago campus network. As a result, people using those ISPs were unable to access on-campus network resources. NSIT network staff have installed a workaround to correct this problem, and will be working with our vendor to obtain a permanent fix for the bug which caused it to begin with.
10-23-2008 10:19 AM Yellow Problems with off-campus network connections
NSIT has received reports that some off-campus users are unable to access network resources on campus. NSIT networking staff are aware of the issue and are working on resolving it.
10-08-2008 9:31 AM Green Webshare Service Restored
At approximately 6:45 AM on Wednesday 8 October, Webshare became unavailable following a routine maintenance reboot. The problem was tracked to a database misconfiguration and service was restored at approximately 9 AM. No data was lost. NSIT is investigating the source of the configuration error and will install processes to prevent this from happening again.
10-08-2008 8:18 AM Yellow WebShare Unavailable

At approximately 6:45AM the WebShare service began experiencing problems and became unavailable for use. NSIT is investigating the issues and is working to restore service as soon as possible.

At this time there is no estimate for restoration of the WebShare service.

10-06-2008 4:27 PM Green Email Delivery Normalized
On the morning of 6 October, a misconfiguration one one of the Junkmail servers began delaying email delivery for a large number of users on campus. The problem was spotted at approximately 10 AM; the configuration error was tracked down and corrected around noon. The resulting rush of email as the Junkmail queue cleared then caused backups on both mailing list servers (listhost and lists). The backlog was processed, and normal email delivery restored, by 1:30 PM; at this point, email is operating normally.
10-06-2008 11:57 AM Yellow Delayed Email Delivery
Around 10 AM 6 October, some users began to experience delays receiving incoming email. The problem has been traced to a backlog on one of the Junkmail servers. NSIT staff is working to trace the source of the backlog and get it cleared.
09-25-2008 11:15 AM Green Problems accessing websites and network services

At approximately 10 AM September 25th, NSIT began receiving isolated reports that some websites and network services were unreachable. Sites affected included cMore, cMail, cnet, the Online Directory, and a number of others.

The problem has been traced to a database problem in our local Domain Name Service (DNS), which serves as a "directory lookup" to help computers find other computers on the network. DNS was restored to an earlier state by approximately 11:15 AM, which restored the correct entries.

Because of the nature of DNS, this problem did not affect all users equally (many people are not experiencing it at all). The fix will also not work "instantly" -- some people may continue to see problems for an hour or so after the DNS restore is complete, while the corrected information propagates.

09-12-2008 9:29 AM Green xMail Service Operating Normally
As of 6:30AM, Friday, September 12, the xMail security certificate update is complete. The xMail service should be operating normally and clients should no longer be experiencing interrmittent warnings about "unable to establish a secure connection...".
09-11-2008 12:43 PM Yellow xMail Service Interruption

Clients using xMail may experience a brief interruption in service, no longer than a few minutes, between 2:00AM and 6:30AM on Friday, September the 12th. This interruption in service will be occurring so that an xMail security certificate can be replaced, which is presently causing some xMail clients to receive interrmittent warnings.


(Read More...)
09-07-2008 8:30 PM Green Central Disk Storage and Gargoyle Maintenance Complete
As of 8 p.m. Sunday, September 7, maintenance on the central disk storage system and Gargoyle and its related systems is complete. All systems should be operating normally, and we will continue to monitor their performance. If you encounter any problems, contact us at 4-TECH (773-834-8324).
09-07-2008 12:40 AM Red Scheduled Maintenance for Gargoyle and the Central Disk Storage System
Gargoyle and related systems are scheduled for maintenance and will be unavailable with no access on Sunday, September 7, from 8 a.m. through approximately 7 p.m. Additionally, from 9 a.m. until 5 p.m., NSIT will be performing maintenance on its central disk storage system. This maintenance involves a large number of updates on dozens of pieces of hardware; as a result, there is a risk that services which depend on central disk storage may slow down or become unavailable during this period. Almost every system NSIT runs could be affected, including (but not limited to): Bulkmail, Chalk, Events calendaring, Griffin, Lists and listhost, Tank fileshares, and WebShare. Primary email services (cMail, xMail, and Junkmail) should not be affected.
09-05-2008 2:02 PM Yellow Central Disk Storage Maintenance
On Sunday, September 7 from 9 a.m. to 5 p.m., NSIT will be performing maintenance on its central disk storage system. This maintenance involves a large number of updates on dozens of pieces of hardware; as a result, there is a risk that services which depend on central disk storage may slow down or become unavailable during this period. Almost every system NSIT runs could be affected, including (but not limited to): Bulkmail, Chalk, Events calendaring, Griffin, Lists and listhost, Tank fileshares, and WebShare. Additionally, cMore, Gargoyle, and other student systems will be taken down entirely from 8 a.m. until 7 p.m. on Sunday, September 7 for extended maintenance. Primary email services (cMail, xMail, and Junkmail) should not be affected.
08-28-2008 7:58 PM Green FindIT Restored
FindIT, NSIT's service catalog, is available again as of 8 PM Thursday, 28 August. The new configuration should be more resistant to high load conditions.
08-28-2008 4:54 PM Yellow FindIT Unavailable
FindIT, NSIT's service catalog, has been made unavailable by an unexpected high load. The usage spike began around 2 PM Thursday, 28 August 2008. As of 5 PM, NSIT staff is working with the vendor behind FindIT to patch the service so it can handle the higher load. We hope to have it available again by 7 PM.
08-27-2008 9:03 AM Green Brief Webmail Outage
On Wednesday, 27 August, a routine update to the "look-and-feel" of webmail.uchicago.edu did not apply properly on one of the front-end webmail servers. As a result, some people were unable to log in between 8:30 and 8:45 AM. The problem server was removed from rotation as soon as the problem was discovered, so the update can be re-applied.
08-13-2008 5:39 PM Green xMail service restored

At approximately 5 PM August 13th, one of the 28 databases containing xMail mailboxes became disconnected from the service. As a result, users whose mailboxes were on that database were unable to access email. The database was successfully reconnected at approximately 5:30 PM. NSIT staff are researching the root cause of the disconnect to ensure it will not recur.

08-13-2008 5:11 PM Red xMail Database Problems
One of the 28 databases supporting Exchange email has become disconnected from the service. As a result, users on that database are currently unable to connect to xMail. Other xMail users should not be having any problems. NSIT systems staff are working to restore the connection.
08-13-2008 9:06 AM Green Tank fileshares, Webshare restored

A regularly scheduled upgrade to our disk infrastructure caused (Storage Area Network, or SAN) caused a number of services to go down on the morning of August 13th. Services affected included the tank1 and tank4 Windows fileshares, Webshare, and Bulkmail.

As of 9 AM, all services have been restored. If you are still having problems with your tank fileshare, restarting your machine should restore access.

NSIT will be working with IBM to discover why the upgrade had this effect and ensure it does not happen again.

08-13-2008 8:48 AM Red Disk problems on tank fileshares, other services

As of 8:15 AM Wednesday August 13th, a problem with the disk infrastructure has brought down several NSIT services, including:

Tank fileshares
Webshare
Bulkmail
Lists

Other systems may also be affected. NSIT systems staff are working to restore service.

08-11-2008 9:15 PM Green network connectivity restored
Network connectivity to Gleacher, 401 N. Michigan and wireless on south campus was restored shortly after 8:30pm.
08-11-2008 7:55 PM Yellow loss of connectivity on campus wireless, all of Gleacher and 401 S Michigan
The data center at 1155 suffered a loss of connectivity between 6 and 6:45pm on 8/11/08. Wireless on the south of campus and connectivity to the Gleacher Center and 401 S Michigan have not yet been restored at 7:45, but we expect to do so by 9pm.
08-05-2008 5:29 PM Green Power Restored to Campus
As of 5:30pm, power has been restored to most of the campus buildings affected by last nights storm.
08-05-2008 7:55 AM Yellow Network down on much of campus due to pwer problems
Large portions of the south side of campus are without network connectivity beginning just before 4am, presumably as a result of thunderstorms taking out power. Some buildings which have power do not have network connectivity because the buildings which feed their network do not have power. We have no word from ComEd regarding when power might be restored.
07-14-2008 10:35 PM Green FindIT Service Restored
FindIT, NSIT's service catalog database, was brought down at approximately 4 PM July 14th by a large-scale power outage in Vancouver, where the service provider is located. Service was restored at approximately 8 PM when the provider was able to transition to emergency power.
07-14-2008 5:23 PM Yellow FindIT Service Down
FindIT, NSIT's catalog of available services, is currently unavailable. NSIT is contacting the vendor to resolve the outage and restore service as soon as possible.
06-20-2008 1:48 PM Green South Campus Phone Service Restored

As of approximately 1:00PM all phone services that were impacted by this morning's fiber cable cut have been restored via a re-routing of the infrastructure.

06-20-2008 9:17 AM Red South Campus Phone Service Outage

11:15AM UPDATE: The fiber is being repaired and for some users service has been restored. However, there remains much to be done and because of the nature of the connections, users may find that their phone remains out while adjacent ones are back in service. We ask staff to please refrain from submitting trouble tickets at this time. Once service has been fully restored, notice will be posted here. Any remaining concerns can then be identified.

10:15AM UPDATE: The fiber cut is still being investigated for repair - no estimate for restoration of full service is available at this time. 4-TECH, 2-9100 and 2-1234 have been rerouted and now have limited capacity for inbound calls.

At approximately 8:20 this morning, portions of the south campus found themselves without University phone services. A fiber communications cable feeding this area was accidentally cut through, curtailing phone service to a number of buildings south of 60th Street. A number of NSIT services, particularly 4-TECH, 2-9100 and 2-1234 University Operators are among those affected. These services are being rerouted in an attempt to answer inquiries arriving about this service outage, but calls may be delayed until services are fully restored.

As further information becomes available, we'll post updates to this notice.

05-20-2008 8:46 AM Green Systems normal
As of 8:45 AM Tuesday, 20 May, NSIT systems are operating normally.
05-19-2008 3:52 PM Yellow Slow email performance

On the morning of Monday, 19 May, a rogue process flooded one person's mailbox with tens of thousands of messages in a short period of time. This caused a backup in email delivery and problems logging into email for approximately a quarter of our users.

As of 3:30 PM, the excess messages have been cleared out, but the system is still catching up with the accumulated backlog. As a result, people may still see email problems into the evening.

NSIT is investigating ways of preventing this kind of problem from recurring.

05-19-2008 12:00 PM Yellow Email Performance
NSIT email is experiencing reduced performance. Accessing email may be very slow (or unavailable). As additional information is available it will be posted here.
05-08-2008 9:29 AM Green Gargoyle System back up
NSIT has rectified the system issues from this morning.

The application is back up and available as normal

Once again, we appreciate your patience and understanding.

05-08-2008 8:57 AM Red Gargoyle System Outage
As of around 8am this morning, the Gargoyle system experienced problems and is currently unavailable.

NSIT staff are working diligently to resolve the problem.

Further updates will be provided as the situation becomes clear.

Thank you for your patience and understanding, as we work to stabilize the application

04-25-2008 2:23 PM Green Gargoyle Performance/Outage Issues
We have isolated the recent performance/outage problems to the running of some queries which produce a report output directly into MS Excel.

It seems that after the query has run on the database, the act of importing records directly into an Excel spreadsheet on the Gargoyle user's system, causes the Webserver to exceed the maximum session memory capacity imposed by the operating system. This is a Microsoft limit and is not reconfigurable.

This has only become apparent recently as reports of a larger nature are being run.

The Gargoyle team has made arrangements with the small subset of users who require these reports, for them to be run externally from Gargoyle.

Consequently, we do not anticipate any further system problems, and if you have not been approached individually, you may assume you can utilize the system as normal.

Once again, the Gargoyle team would like to thank you for your continued patience and understanding as we endeavor to provide a stable and efficient application environment.

04-24-2008 3:31 PM Yellow Gargoyle System performance Issues
We are experiencing some intermittent performance issues whereby a particular job causes the system to overload. we have asked the business area that runs this task to refrain, which seems to let the system stay stable.

Gargoyle is generally up and available, but be aware it may encounter random system outages

Further updates will be provided as they develop.

04-24-2008 9:08 AM Green Windows Login Problems resolved.
A domain login controller in the ADLOCAL Windows domain began refusing connections earlier this morning (Thursday, 24 April 2008). This caused a random subset of administrative users to be unable to log in to their Windows desktop machines. NSIT systems staff have fixed the problem; all login services should now be operating normally.
04-24-2008 8:49 AM Red Windows Login Problems
The NSIT Support Line is receiving numerous reports of problems logging on to ADLOCAL, the centrally administered Windows domain. Commonly, people are unable to log into their desktop Windows machine, and/or receive an error message about using cached credentials. NSIT systems and support staff are working to isolate and resolve the problem.
04-14-2008 1:35 PM Green Gargoyle outage update
The current problems on Gargoyle are resolved. Users should be able to perform normal functions. If you encounter any further problems, please send an e-mail to gargoylehelp@listhost.uchicago.edu. System administrators will be monitoring Gargoyle for any further problems.
03-17-2008 10:03 AM Yellow Password-stealing scam on campus

Over the weekend of March 15th, a large number of people on campus received email purporting to be from "THE UCHICAGO.EDU TEAM". This email asked the recipient to email back with their password to prevent their account from being "deactivated". This message did NOT come from NSIT or any University authority; it was a "phishing scam" used to steal people's passwords and accounts. If you responded to this message, you should change your password immediately. (Network Security is also working to identify accounts stolen by this scam and proactively lock them to prevent misuse.)

This is not the first time this scam has hit the campus, and will not be the last. You should be aware that, by policy, NSIT will NEVER contact you to ask for your password, either by phone or by email. It is also explicitly against University policy to send your password to anyone via email. If you should receive such a request via email, you should delete it without responding. For more information, see the Safecomputing website's article on Phishing at http://safecomputing.uchicago.edu/phishing/.

View all events for IT Services
(This list does not show expired events. They will be displayed when viewing all events.)