|
10-30-2009 9:38 AM |
Green
|
Find IT Restored
At approximately 9 a.m. CDT on 30 October 2009 Find IT has been restored to normal functionality. |
| 10-29-2009 4:40 PM |
Yellow |
Find IT Unavailable
Find IT, NSIT's service catalog, is currently unavailable. We are in communication with the vendor to resolve this problem. |
| 10-27-2009 10:19 PM |
Green |
VPN Logins Restored
On the afternoon of Tuesday, October 27th, NSIT began receiving reports of isolated problems logging in to the University VPN. The problem was tracked to a missing attribute on the users' CNetIDs. Further investigation revealed that approximately 10% of our users were incorrectly missing this attribute. The root cause of the problem was corrupted data in the database managing the VPN access group. As of 6:15 PM, this corruption has been cleaned out, correct information restored, and all eligible users can once again access the VPN. |
| 10-27-2009 5:27 PM |
Yellow |
Email Issues Resolved; VPN issue still ongoing
Email being sent to @uchicago addresses is rotated through one of three mail servers for delivery. At approximately 2 PM Tuesday, 27 October, a cache problem on one of these servers caused incoming messages routed through that server to be returned to their senders. As of 5:15 PM, the cache problem has been cleared, and normal email delivery has been restored. This problem was unrelated to problems some users are having logging on to the VPN; NSIT staff are still working on that issue. |
| 10-27-2009 4:59 PM |
Yellow |
Isolated VPN Login Problems
NSIT has received complaints that some users are unable to log in to the University VPN service. The issue has been traced to a missing attribute on the users' CNetIDs; further investigation suggests that this attribute is missing for around 10% of our users. NSIT staff are currently working to restore this attribute and normal VPN access. This problem is unrelated to ongoing problems with email delivery to @uchicago.edu addresses. |
| 10-27-2009 4:53 PM |
Yellow |
Sporadic Bounces to @uchicago.edu addresses
NSIT has received reports of sporadic problems with email delivery; some messages sent to @uchicago.edu addresses are being returned to senders. The problem has been traced to one of the three incoming email servers we have in rotation, and NSIT staff is currently working to resolve the issue. |
| 10-08-2009 2:50 PM |
Green |
Resolved: Slow Performance on cMail
NSIT Systems Engineers identified the cause for the slowness impacting a portion of the cMail system which began earlier this afternoon. An automated backup system erroneously began backing-up the storage system for one of the backend mail systems, which degraded the performance for users on that system. NSIT Systems Engineers cancelled the erroneous backup job at approximately 1:45PM. The impacted mail system has cleared its backlog and resumed normal operation. |
| 10-08-2009 2:03 PM |
Yellow |
Updated: Slow Performance on cMail
NSIT Systems Engineers have identified the cause for the slowness impacting a portion of the cMail system which began earlier this afternoon. An automated backup system erroneously began backing-up the storage system for one of the backend mail systems, which degraded the performance for users on that system. NSIT Systems Engineers cancelled the erroneous backup job at approximately 1:45PM. Individuals may experience degraded performance as the impacted mail system returns to normal operation. |
| 10-08-2009 1:35 PM |
Yellow |
Slow Performance on cMail
NSIT is aware of recent performance issues on cMail, our central standard email system. This tends to take the form of long delays or hangs when logging in or accessing email for some, but not all users. NSIT Systems Engineers are investigating the issue and we will post more information as it becomes available. |
| 09-13-2009 11:20 PM |
Green |
Chalk Restored
Chalk, the University's central learning management system, experienced a filesystem failure during standard maintenance on the morning of Sunday, 13 September. As a result, Chalk was unavailable for most of the day Sunday while NSIT staff attempted to recover the drive; in the end, they were forced to reformat and restore from the Friday night backup. As of 11:15 PM, Chalk is once again fully functional. Files uploaded on Saturday will need to be re-uploaded; other changes (e.g. forum postings or tests/surveys) should be fine, as they would not have been on the affected drive. |
| 09-13-2009 6:16 PM |
Red |
Update (6:15 PM): Chalk Outage
Chalk, the University's central learning management system, experienced a filesystem failure during standard maintenance on the morning of Sunday, 13 September. As of 6:15 PM, NSIT staff are restoring Chalk from last night's backup. Chalk is expected to be back up before Monday morning. |
| 09-13-2009 3:31 PM |
Red |
Update: Chalk Outage
Chalk, the University's central learning management system, experienced a filesystem failure during standard maintenance on the morning of Sunday, 13 September. As of 3:30 PM, Chalk is still unavailable; NSIT staff are working with vendor support to recover the system. |
| 09-13-2009 11:43 AM |
Red |
Chalk Outage
As of 11:30 AM 13 September, Chalk, the University's central learning management system, is unavailable due to problems with its underlying filesystem. NSIT staff are working to resolve the problem. |
| 09-04-2009 2:15 PM |
Green |
cTV Maintenance Complete
As of 2:00 p.m. on Friday, September 4, 2009, the scheduled cTV maintenance is complete and cTV is available for viewing.
|
| 09-03-2009 11:57 PM |
Yellow |
cTV Maintenance
cTV will be unavailable on Friday, September 4, 2009 due to vendor-required maintenance. |
| 08-27-2009 2:34 PM |
Green |
Find IT Restored
At approximately 1 p.m. CDT on 27 August 2009, Find IT has been restored to normal functionality. |
| 08-27-2009 11:19 AM |
Yellow |
Find IT Unavailable
Find IT, NSIT's service catalog, is currently unavailable. We are in communication with the vendor to resolve this problem. |
| 08-25-2009 4:06 PM |
Green |
Systems Restored
At approximately 3:00 am CDT on 25 August 2009, there was an isolated storage system failure that impacted a number of smaller servers including formulator.uchicago.edu, wiki.projectbamboo.org, and the organization and help request forms used by Chalk. The issue was traced to a configuration problem, which has now been remedied; as of 4 PM, all systems should now be running normally. |
| 08-25-2009 12:24 PM |
Yellow |
FindIT restored; other systems still down
As of approximately 12 noon, 25 August, FindIT has been restored to normal functionality. However, the original incident, involving an isolated storage system failure that impacted a number of smaller servers including formulator.uchicago.edu, wiki.projectbamboo.org, and the organization and help request forms used by Chalk, is still being worked on. This particular outage is limited only to these minor systems; Chalk and other major systems are unaffected. NSIT staff are working to restore these services as soon as possible. |
| 08-25-2009 9:42 AM |
Yellow |
Storage Failure issues on some systems; FindIT also down
At approximately 3:00 am CDT on 25 August 2009, there was an isolated storage system failure that impacted a number of smaller servers including formulator.uchicago.edu, wiki.projectbamboo.org, and the organization and help request forms used by Chalk. This particular outage is limited only to these minor systems; Chalk and other major systems are unaffected. NSIT staff are working to restore these services as soon as possible.
In an unrelated incident, FindIT, NSIT's service catalog, is down. We are in communication with the vendor to resolve this problem. |
| 08-25-2009 9:04 AM |
Yellow |
Storage Failure Problems on several systems
At approximately 3:00 am CDT on 25 August 2009, there was an isolated storage system failure that impacted a number of smaller servers including formulator.uchicago.edu, wiki.projectbamboo.org, and the organization and help request forms used by Chalk. This particular outage is limited only to these minor systems; Chalk and other major systems are unaffected. NSIT staff are working to restore these services as soon as possible.
|
| 08-14-2009 8:52 AM |
Green |
Resolved: xMail Service Issues
NSIT staff isolated the cause of the instability in the xMail service at 6:30pm on Thursday, August 13th, and took action to repair the xMail service. As of this notice, the xMail service is once again fully operational and all individuals should have access. We remain in contact with the vendor and are closely monitoring the servers to ensure that this problem does not recur. |
| 08-13-2009 4:32 PM |
Yellow |
xMail Service Issues - Update
Beginning late in the afternoon of Wednesday, 12 August, two of the front-end servers to xMail, NSIT's central Exchange service, began experiencing problems, which have been growing progressively worse. This issue causes sporadic disconnects or login problems for people trying to access xMail using IMAP (e.g. with Thunderbird) or over the web. As of 4:30 PM Thursday, 13 August, the problem has expanded to include issues sending mail with Outlook -- Outlook clients can see existing mailboxes and incoming email, but cannot consistently send mail. NSIT is continuing to work with the vendor to resolve this issue. |
| 08-13-2009 2:50 PM |
Yellow |
Service Issues on xMail
Beginning late in the afternoon of Wednesday, 12 August, two of the front-end servers to xMail, NSIT's central Exchange service, began experiencing problems, which have been growing progressively worse. This issue causes sporadic disconnects or login problems for people trying to access xMail using IMAP (e.g. with Thunderbird) or over the web. Users with a standard Outlook configuration may see occasional slow performance. NSIT is currently working with the vendor to resolve the problem. |
| 08-10-2009 8:59 AM |
Green |
Early Monday cMail Outage
A significant portion of the cMail system lost the ability to authenticate clients from approximately 1:00AM until 7:00AM, early Monday morning. The affected components of the service were restarted at 6:15AM and access was restored to all clients shortly thereafter.
NSIT Staff are consulting with the mail system vendor to determine why this interruption in service occurred. |
| 08-05-2009 7:09 PM |
Green |
Gargoyle Student System - Service restored
Service to the Gargoyle Student System was restored at approximately 6:30pm. The application is now available for use. NSIT will be conducting a root cause analysis to identify and understand the cause of this outage.
|
| 08-05-2009 4:08 PM |
Red |
Gargoyle Student System Disruption
The Gargoyle system is experiencing a problem, we are aware of it and system administrators are working to find what is causing this and to resolve it as quickly as possible.
Please logoff of the Gargoyle system and remain off the system until further notice. We will send out another email to all users when it is OK to use Gargoyle again.
We thank you in advance for your patience and understanding at this time.
|
| 07-29-2009 1:43 PM |
Green |
Resolved cMail Outage
While conducting an upgrade on the storage servers that support cMail, an unforeseen configuration conflict caused part of the cMail service to fail. With on-site vendor assistance, the configuration conflict was resolved and the cMail service was restarted successfully shortly after 1PM. No email was lost as a result of this outage. |
| 07-29-2009 12:51 PM |
Red |
cMail Outage
At approximately 12:30 PM Wednesday, 29 July, two of the mail storage units on cMail, the University's central email system, stopped responding. As a result, approximately half of the people using cMail are currently unable to check their mail. NSIT systems staff are currently working with the vendor (who is on-site) to resolve the problem. |
| 07-21-2009 12:30 PM |
Green |
CNN Restored on cTV
The faulty hardware component has been replaced and CNN is available. The cTV service is now functioning normally. |
| 07-20-2009 10:32 AM |
Yellow |
cTV CNN Problem
The component of the cTV service, which provides CNN, is currently experiencing a hardware problem. As a result, Channel 100 (CNN) is not available at this time. NSIT is working with the vendor to obtain replacement hardware, which is currently expected to arrive tomorrow morning. CNN should be available again before NOON tomorrow (7/21). |
| 07-13-2009 10:12 AM |
Green |
Resolved: Bouncing Email on lists
Reminder: Any email sent to a list on lists.uchicago.edu during the ~35 minutes of the outage on Friday should have been returned to the individual who sent it; they will need to re-send the message to make sure it goes through. |
| 07-10-2009 12:20 PM |
Yellow |
Bouncing Email on lists
On 10 July, at approximately 11:40 AM, a configuration error on lists.uchicago.edu began causing email sent to mailing lists to bounce. The problem was discovered and fixed at approximately 12:15 PM. Any email sent to a list on lists.uchicago.edu during the ~35 minutes of the outage should have been returned to the individual who sent it; they will need to re-send the message to make sure it goes through. |
| 07-01-2009 12:10 PM |
Green |
Resolved: Authentication Problems
Beginning around 8:45am on July 1, many people on campus became unable to authenticate to the University wireless network. Others were also unable to access webspace.uchicago.edu. The problem was traced to an error that occurred when propagating eligibility information to our authentication systems. As of 11:45am, NSIT has restored all access by repopulating the eligibility fields. |
| 07-01-2009 10:50 AM |
Yellow |
Wireless Authentication Problem -- Update
On the morning of July 1, a large number of people on campus became unable to authenticate to the University wireless network. The problem has been traced to an error which occurred when propagating wireless eligibility to our authentication system. Systems staff are currently repopulating the eligibility fields; people will be able to log in again as the correct data propagates. Unfortunately, the process is taking longer than we anticipated; we currently project that the problem will be resolved by 1 PM. The error did not affect any service other than wireless access. |
| 07-01-2009 9:57 AM |
Yellow |
Wireless Authentication Problems
A large number of people on campus are currently unable to authenticate to the University wireless network. The root of the problem has been traced to an error which occurred when propagating wireless eligibility to our authentication system. Systems staff are currently repopulating the eligibility fields; people will be able to log in again as the correct data propagates. The problem should be entirely resolved by 10:30 AM. The error did not affect any service other than wireless access. |
| 06-15-2009 10:48 AM |
Green |
NSIT service outage on 6/11/2009 that impacted campus websites
This message is a follow-up to the outage on Thursday, June 11 that severed access to over 1,000 campus websites from about noon to 5:30 pm. We want to share with you, what happened, the solution process, and plans to ensure that this type of incident does not re-occur.
(Read More...)
|
| 06-11-2009 5:44 PM |
Green |
Resolved: Web hosting environment outage
As of 5:35pm on 6/11, NSIT personnel have implemented a solution to restore the web hosting environment's full functionality. All sites are once again accessible. |
| 06-11-2009 2:33 PM |
Red |
Updated: Web hosting environment outage
As of noon on June 11th, NSIT began experiencing DNS-related problems with one of the web hosting environments. Many websites, primarily those hosted on webspace.uchicago.edu, are presently unavailable. The source of the problem has been identified and NSIT personnel continue to work in conjunction with an outside vendor. At this time, we do not anticipate a resolution before the close of business today, 6/11. |
| 06-11-2009 1:03 PM |
Red |
Web hosting environment outage
NSIT is presently experiencing problems with one of the web hosting environments. Many websites, primarily those hosted on webspace.uchicago.edu, are presently unavailable. The source of the problem has been identified and NSIT personnel expect to resolve the problem before 2pm. |
| 05-14-2009 1:40 PM |
Green |
USITE, cMail Performance Issues Resolved
After consultation and service calls from both hardware and software vendors, the cluster of servers supporting the USITE workstations is again operating at 100% capacity. Service was not interrupted during the course of this incident. Additionally, the vendor-installed patch on the morning of 13 May appears to have resolved performance issues for cMail users; if you are still seeing hangs or slow performance using cMail, please contact the NSIT Support Line at support@uchicago.edu or 834-TECH. |
| 05-11-2009 3:52 PM |
Yellow |
Slow performance on USITE Workstations
We are experiencing a hardware problem with one of the four servers which provide USITE users with their Windows environment. We are investigating the problem to determine the cause and severity. In the mean time, the faulty system has been removed from the cluster. All system users may notice slightly slower response time from their workstation during peak utilization hours, when load on the remaining members of the cluster will increase.
|
| 05-07-2009 4:40 PM |
Yellow |
Slow Performance on cMail
NSIT is aware of recent performance issues on cMail, our central standard email system. This tends to take the form of long delays or hangs when logging in or accessing email. We have spoken with the vendor, and they are providing us with a patch which they say will address the issue; they will be installing this patch during our normal early morning maintenance period on Wednesday, 13 May. In the meantime, we have noted that the problem seems to be much worse for people using webmail.uchicago.edu than for people using desktop clients (such as Thunderbird), so you may see improved performance if you try a different client; instructions for configuring desktop clients can be found at
http://nsit.uchicago.edu/docs/email/configuration/ |
| 05-04-2009 9:45 AM |
Green |
Harper back up; home still read-only
At approximately 2:45 PM Thursday, 30 April, a disk failure began causing problems for users of harper, NSIT's central interactive Unix system. NSIT systems staff were forced to do an emergency shutdown to deal with the failure at approximately 3:30 PM, which also interrupted service for home.uchicago.edu, NSIT's central personal web page service. As of 9 AM Monday, 4 May, harper is back up and running normally with files restored from the backup done on the night of 29 April. Web pages on home.uchicago.edu can be read normally over the web, but they cannot be updated; we are working to make these two services more independent of each other before fully restoring write access to home pages. We will contact home page owners shortly with more information. |
| 05-01-2009 5:31 PM |
Yellow |
Harper partially restored; home.uchicago.edu still read-only
At approximately 2:45 PM Thursday, 30 April, a disk failure began causing problems for users of harper, NSIT's central interactive Unix system. NSIT systems staff were forced to do an emergency shutdown to deal with the failure at approximately 3:30 PM. As of 5 PM Friday 01 May, harper's user files have been restored to their status as of Wednesday night, 29 April, and logins to harper have been turned back on. Work to stablize the system is continuing, however, and the machine will be periodically unavailable over the weekend due to reboots. Additionally, although home.uchicago.edu is serving files out to the web, it is not possible to update or change pages on home (via public_html) at this time. More information will be posted here as it becomes available. |
| 05-01-2009 9:27 AM |
Yellow |
Harper unavailable; home.uchicago.edu disruptions possible
At approximately 2:45 PM Thursday, 30 April, a disk failure began causing problems for users of harper, NSIT's central interactive Unix system. NSIT systems staff were forced to do an emergency shutdown to deal with the failure at approximately 3:30 PM, and the system has been unavailable since that time. As of 9 AM Friday, 01 May, NSIT systems staff have succeeded in recovering data from the lost disk, but are still working to get the disk array back online; this may lead to brief disruptions in the availability of web pages on home.uchicago.edu. We hope to have harper restored by noon. People who access their email using harper should be able to access their new email over the web (at https://webmail.uchicago.edu/ ) in the interim. |
| 04-30-2009 5:29 PM |
Yellow |
home restored; harper still unavailable
At approximately 2:45 PM Thursday, 30 April, a disk failure began causing problems for users of harper, NSIT's central interactive Unix system. NSIT systems staff were forced to do an emergency shutdown to deal with the failure at approximately 3:30 PM; this shutdown also had the effect of shutting down home.uchicago.edu, the central home page service for the University. As of 5:30 PM, home.uchicago.edu is back up in a read-only state (i.e. it is not be possible to update pages, but they are viewable over the web). Systems staff are continuing to work on harper, but we do not expect it to be available again before tomorrow. People who use harper for email should be able to access their new mail over the web (at https://webmail.uchicago.edu/ ) until harper is restored. |
| 04-30-2009 4:50 PM |
Yellow |
home, harper disk failure
At approximately 2:45 PM Thursday, 30 April, a disk failure began causing problems for users of harper, NSIT's central interactive Unix system. NSIT systems staff were forced to do an emergency shutdown to deal with the failure at approximately 3:30 PM; this shutdown also had the effect of shutting down home.uchicago.edu, the central home page service for the University. Work is currently under way to restore these systems. We expect to have home.uchicago.edu back up later this evening in a read-only state (i.e. it will not be possible to update pages, but they will be viewable over the web). We do not expect harper to be available again before tomorrow. People who use harper for email should be able to access their new mail over the web (at https://webmail.uchicago.edu/ ) until harper is restored. |
| 04-30-2009 12:22 PM |
Green |
cMore Restored
Over the night of 29 April 2009, NSIT began receiving reports of login problems on the cMore application. As of 12:15 PM 30 April, a workaround has been put in place and normal logins have been restored. The incident has been tentatively traced to a database access problem, but NSIT staff are continuing to work on a permanent resolution to the underlying issue. |
| 04-30-2009 10:41 AM |
Red |
cMore Unavailable
Over the night of 29 April, NSIT became aware of problems with cMore logins. As of 9 AM Thursday, April 30th, all cMore users were experiencing hangs on login. NSIT has disabled logins on cMore while we investigate this issue. More updates will be posted as they become available.
|
| 04-20-2009 7:08 AM |
Green |
All services restored
At approximately 6 AM Monday, 20 April, a number of machines in NSIT's "virtual" machine service stopped responding. As a result, services running in part or in whole on those machines may be running slowly or may not be available at all. Affected services include Griffin, Gargoyle and other student systems, and the tank1 central Windows file share, among others. As of 7 AM, all the systems in question have been restarted, and all services appear to be operating normally. NSIT systems staff will continue to investigate the source of the problem to ensure it does not happen again. |
| 04-20-2009 6:36 AM |
Red |
Service issue for Griffin, Student Systems, tank1
At approximately 6 AM Monday, 20 April, a number of machines in NSIT's "virtual" machine service stopped responding. As a result, services running in part or in whole on those machines may be running slowly or may not be available at all. Affected services include Griffin, Gargoyle and other student systems, and the tank1 central Windows file share, among others. As of 6:30 AM, NSIT systems staff have restarted the systems in question and are monitoring to ensure that systems come back online. |
| 04-02-2009 12:29 PM |
Green |
xMail Service Restored
On the morning of Thursday, April 2nd, 2009, a security filter caused a number of xMail users to have trouble accessing their email using Outlook. The problem filter was identified and corrected shortly after noon. NSIT systems staff are investigating the origin of the filter issue and will insert processes to insure this does not happen again. |
| 04-02-2009 11:21 AM |
Yellow |
xMail Login Problems
On the morning of Thursday, April 2nd, some users began reporting trouble logging in to xMail using Outlook. Some of these users are able to connect after repeated tries, others are unable to log in at all. We have also received reports of timeouts during continuing sessions. NSIT systems staff is aware of the issue and working to resolve it. As a workaround, we recommend logging in via the xmail.uchicago.edu web interface. |
| 03-24-2009 1:30 PM |
Green |
xMail Performance Issues Resolved
NSIT Staff working in conjunction with an external vendor found and replaced a piece of problematic hardware. Replacement of the problematic hardware resolved the performance issues with the xMail service as of approximately 1:15PM. |
| 03-24-2009 9:22 AM |
Yellow |
xMail Slowness
NSIT has observed an increasing slowness with the xMail service. We will be taking pre-emptive action at approximately 9:20AM to ameliorate this slowness with the possibility of a potential unscheduled restart.
10:25AM UPDATE: Actions taken by NSIT Engineers have improved xMail performance, but we continue to monitor the situation and may take further action if the slowness redevelops. |
| 02-06-2009 3:15 PM |
Green |
Email problems resolved
At approximately 2:30 PM Friday 06 February, one of the email stores on cMail began having severe performance problems. Anyone with email stored on that part of the system (approximately a quarter of the cMail users) was unable to log into or access their email. The issue was resolved at approximately 2:50 PM by a restart of the mail store in question. No email was lost, and as of 3 PM, normal service has been restored. NSIT systems staff are working with the vendor to identify the root cause of the problem and ensure it does not recur. |
| 02-06-2009 2:47 PM |
Yellow |
Problems accessing email
One of the mail stores on cMail, NSIT's primary email service, is experiencing problems. Any users who have mail on that store (approximately a quarter of the user base) may have problems logging into or accessing email, or may see severe performance problems. NSIT systems staff are working to identify and resolve the issue. |
| 12-17-2008 9:33 PM |
Green |
Voicemail System normal
The university voicemail system was returned to full functionality at approximately 7:15PM this evening. |
| 12-17-2008 4:47 PM |
Red |
Voicemail system announces mailbox is full
At about 4:30pm today, the university's voicemail system began reporting users' mailboxes were full and no message could be left by the caller. The matter has been escalated to technical staff who are currently investigating the matter. Updates and resolution will be posted as they become available this evening.
As of 6:30pm, the problem is still under active investigation. Further updates shall follow. |
| 11-20-2008 9:08 AM |
Green |
Systems normal
We are not aware of any system problems at this time. If you are still receiving "Relaying Denied" errors when sending email from off-campus, you should update your email configuration; see http://support.uchicago.edu/authsmtp |
| 11-18-2008 5:18 PM |
Yellow |
"Relaying Denied" errors sending email from off-campus
On the morning of Wednesday, November 19th, NSIT will block off-campus access to the central outbound email server, smtp.uchicago.edu. After that time, anyone who is still using an email client configured to use that server will receive "Relaying Denied" errors when trying to send email from off-campus. The fix is to update the email settings to use a new, authenticated outbound mail server, authsmtp.uchicago.edu. For instructions, see http://support.uchicago.edu/authsmtp |
| 11-10-2008 12:15 PM |
Green |
Voicemail system normal
The voicemail integration has been resolved; users may again directly access their voicemail using the desk phone "Mail" button.
Posted 12:15PM, Monday, November 10, 2008 |
| 11-10-2008 11:04 AM |
Yellow |
Voicemail access
At about 10:30am, the load on the campus voicemail system spiked and has temporarily altered the way campus users who are accustomed to pressing the "Mail" button from their desk phone may access voicemail. Campus callers are now presented with the greeting customarily given to off-campus callers. To circumvent this temporary problem, users may press the * (star) key and re-enter their 5-digit mailbox number (generally the same as the phone number).
Posted 11:00AM, Monday, November 10, 2008 |
| 10-27-2008 12:13 PM |
Green |
Systems Normal
As of noon, 27 October, there are no signs that the Windows security problem Microsoft revealed last Thursday is being exploited on campus. However, we still encourage anyone running any version of Windows invoke "Windows Update" (or "Microsoft Update") and make sure all critical security patches are applied. For more information on this security issue, see http://www.microsoft.com/technet/security/bulletin/MS08-067.mspx
|
| 10-23-2008 7:08 PM |
Red |
Critical Microsoft Security Patch
On Thursday 23 October, Microsoft issued an emergency security bulletin regarding a vulnerability in Windows XP, Windows Vista, and Windows Server products which could allow a remote user to take over local machines and/or use those machines to propagate a worm or virus. Given the extreme nature of the vulnerability and Microsoft's own unusually strong reaction to it, NSIT believes that all Windows machines at the University should be patched against this vulnerability as quickly as possible. We are currently working to update on-campus desktops supported by NSIT. If you use a Windows laptop or have a Windows computer at home, we urge you to update your machine immediately, by going under the Windows or Start menu to "Windows Update" or "Microsoft Update". Look for the critical security update for KB958644. The original Microsoft security bulletin, and instructions for manually patching your machine, can be found at: http://www.microsoft.com/technet/security/bulletin/MS08-067.mspx |
| 10-23-2008 4:49 PM |
Green |
Emergency update complete
On the afternoon of Thursday, 23 October, Microsoft released a high-priority alert regarding a previously-undisclosed, severe security hole in the Windows Server operating system. In response, NSIT began an emergency update to all its Windows-based services (including xMail, and tank file shares, among others) at approximately 2:30 PM. The updates may have caused slow performance for some users. All services were updated by approximately 4:30 PM, and are now operating normally. For more information about the security issues resolved by this update, see http://www.microsoft.com/technet/security/bulletin/MS08-067.mspx |
| 10-23-2008 2:59 PM |
Yellow |
Emergency updates for xMail, tank fileshares, other services
On Thursday, October 23rd, Microsoft released an alert regarding a severe security hole in current versions of Windows Server, which is the operating system behind xMail, the tank file shares, and several other NSIT services. In response to this alert, NSIT is currently performing an emergency update to apply the newly-released Microsoft patch for this security problem. Although it will not be necessary to disrupt any services while the emergency update is taking place, some users may see slow performance on tank fileshares, xMail, or other services. The emergency update should be complete, and all services restored to normal, by 4 PM. |
| 10-23-2008 2:58 PM |
Green |
Off-campus access issues resolved
Early in the morning of 23 October, a routine update to the network name servers unexpectedly caused a small number of external internet server providers (ISPs) to pick up bad information about the uchicago campus network. As a result, people using those ISPs were unable to access on-campus network resources. NSIT network staff have installed a workaround to correct this problem, and will be working with our vendor to obtain a permanent fix for the bug which caused it to begin with. |
| 10-23-2008 10:19 AM |
Yellow |
Problems with off-campus network connections
NSIT has received reports that some off-campus users are unable to access network resources on campus. NSIT networking staff are aware of the issue and are working on resolving it. |
| 10-08-2008 9:31 AM |
Green |
Webshare Service Restored
At approximately 6:45 AM on Wednesday 8 October, Webshare became unavailable following a routine maintenance reboot. The problem was tracked to a database misconfiguration and service was restored at approximately 9 AM. No data was lost. NSIT is investigating the source of the configuration error and will install processes to prevent this from happening again. |
| 10-08-2008 8:18 AM |
Yellow |
WebShare Unavailable
At approximately 6:45AM the WebShare service began experiencing problems and became unavailable for use. NSIT is investigating the issues and is working to restore service as soon as possible.
At this time there is no estimate for restoration of the WebShare service. |
| 10-06-2008 4:27 PM |
Green |
Email Delivery Normalized
On the morning of 6 October, a misconfiguration one one of the Junkmail servers began delaying email delivery for a large number of users on campus. The problem was spotted at approximately 10 AM; the configuration error was tracked down and corrected around noon. The resulting rush of email as the Junkmail queue cleared then caused backups on both mailing list servers (listhost and lists). The backlog was processed, and normal email delivery restored, by 1:30 PM; at this point, email is operating normally. |
| 10-06-2008 11:57 AM |
Yellow |
Delayed Email Delivery
Around 10 AM 6 October, some users began to experience delays receiving incoming email. The problem has been traced to a backlog on one of the Junkmail servers. NSIT staff is working to trace the source of the backlog and get it cleared. |
| 09-25-2008 11:15 AM |
Green |
Problems accessing websites and network services
At approximately 10 AM September 25th, NSIT began receiving isolated reports that some websites and network services were unreachable. Sites affected included cMore, cMail, cnet, the Online Directory, and a number of others.
The problem has been traced to a database problem in our local Domain Name Service (DNS), which serves as a "directory lookup" to help computers find other computers on the network. DNS was restored to an earlier state by approximately 11:15 AM, which restored the correct entries.
Because of the nature of DNS, this problem did not affect all users equally (many people are not experiencing it at all). The fix will also not work "instantly" -- some people may continue to see problems for an hour or so after the DNS restore is complete, while the corrected information propagates. |
| 09-12-2008 9:29 AM |
Green |
xMail Service Operating Normally
As of 6:30AM, Friday, September 12, the xMail security certificate update is complete. The xMail service should be operating normally and clients should no longer be experiencing interrmittent warnings about "unable to establish a secure connection...". |
| 09-11-2008 12:43 PM |
Yellow |
xMail Service Interruption
Clients using xMail may experience a brief interruption in service, no longer than a few minutes, between 2:00AM and 6:30AM on Friday, September the 12th. This interruption in service will be occurring so that an xMail security certificate can be replaced, which is presently causing some xMail clients to receive interrmittent warnings.
(Read More...)
|
| 09-07-2008 8:30 PM |
Green |
Central Disk Storage and Gargoyle Maintenance Complete
As of 8 p.m. Sunday, September 7, maintenance on the central disk storage system and Gargoyle and its related systems is complete. All systems should be operating normally, and we will continue to monitor their performance. If you encounter any problems, contact us at 4-TECH (773-834-8324). |
| 09-07-2008 12:40 AM |
Red |
Scheduled Maintenance for Gargoyle and the Central Disk Storage System
Gargoyle and related systems are scheduled for maintenance and will be unavailable with no access on Sunday, September 7, from 8 a.m. through approximately 7 p.m. Additionally, from 9 a.m. until 5 p.m., NSIT will be performing maintenance on its central disk storage system. This maintenance involves a large number of updates on dozens of pieces of hardware; as a result, there is a risk that services which depend on central disk storage may slow down or become unavailable during this period. Almost every system NSIT runs could be affected, including (but not limited to): Bulkmail, Chalk, Events calendaring, Griffin, Lists and listhost, Tank fileshares, and WebShare. Primary email services (cMail, xMail, and Junkmail) should not be affected. |
| 09-05-2008 2:02 PM |
Yellow |
Central Disk Storage Maintenance
On Sunday, September 7 from 9 a.m. to 5 p.m., NSIT will be performing maintenance on its central disk storage system. This maintenance involves a large number of updates on dozens of pieces of hardware; as a result, there is a risk that services which depend on central disk storage may slow down or become unavailable during this period. Almost every system NSIT runs could be affected, including (but not limited to): Bulkmail, Chalk, Events calendaring, Griffin, Lists and listhost, Tank fileshares, and WebShare. Additionally, cMore, Gargoyle, and other student systems will be taken down entirely from 8 a.m. until 7 p.m. on Sunday, September 7 for extended maintenance. Primary email services (cMail, xMail, and Junkmail) should not be affected. |
| 08-28-2008 7:58 PM |
Green |
FindIT Restored
FindIT, NSIT's service catalog, is available again as of 8 PM Thursday, 28 August. The new configuration should be more resistant to high load conditions. |
| 08-28-2008 4:54 PM |
Yellow |
FindIT Unavailable
FindIT, NSIT's service catalog, has been made unavailable by an unexpected high load. The usage spike began around 2 PM Thursday, 28 August 2008. As of 5 PM, NSIT staff is working with the vendor behind FindIT to patch the service so it can handle the higher load. We hope to have it available again by 7 PM. |
| 08-27-2008 9:03 AM |
Green |
Brief Webmail Outage
On Wednesday, 27 August, a routine update to the "look-and-feel" of webmail.uchicago.edu did not apply properly on one of the front-end webmail servers. As a result, some people were unable to log in between 8:30 and 8:45 AM. The problem server was removed from rotation as soon as the problem was discovered, so the update can be re-applied. |
| 08-13-2008 5:39 PM |
Green |
xMail service restored
At approximately 5 PM August 13th, one of the 28 databases containing xMail mailboxes became disconnected from the service. As a result, users whose mailboxes were on that database were unable to access email. The database was successfully reconnected at approximately 5:30 PM. NSIT staff are researching the root cause of the disconnect to ensure it will not recur. |
| 08-13-2008 5:11 PM |
Red |
xMail Database Problems
One of the 28 databases supporting Exchange email has become disconnected from the service. As a result, users on that database are currently unable to connect to xMail. Other xMail users should not be having any problems.
NSIT systems staff are working to restore the connection.
|
| 08-13-2008 9:06 AM |
Green |
Tank fileshares, Webshare restored
A regularly scheduled upgrade to our disk infrastructure caused (Storage Area Network, or SAN) caused a number of services to go down on the morning of August 13th. Services affected included the tank1 and tank4 Windows fileshares, Webshare, and Bulkmail.
As of 9 AM, all services have been restored. If you are still having problems with your tank fileshare, restarting your machine should restore access.
NSIT will be working with IBM to discover why the upgrade had this effect and ensure it does not happen again. |
| 08-13-2008 8:48 AM |
Red |
Disk problems on tank fileshares, other services
As of 8:15 AM Wednesday August 13th, a problem with the disk infrastructure has brought down several NSIT services, including:
Tank fileshares
Webshare
Bulkmail
Lists
Other systems may also be affected. NSIT systems staff are working to restore service. |
| 08-11-2008 9:15 PM |
Green |
network connectivity restored
Network connectivity to Gleacher, 401 N. Michigan and wireless on south campus was restored shortly after 8:30pm. |
| 08-11-2008 7:55 PM |
Yellow |
loss of connectivity on campus wireless, all of Gleacher and 401 S Michigan
The data center at 1155 suffered a loss of connectivity between 6 and 6:45pm on 8/11/08. Wireless on the south of campus and connectivity to the Gleacher Center and 401 S Michigan have not yet been restored at 7:45, but we expect to do so by 9pm. |
| 08-05-2008 5:29 PM |
Green |
Power Restored to Campus
As of 5:30pm, power has been restored to most of the campus buildings affected by last nights storm. |
| 08-05-2008 7:55 AM |
Yellow |
Network down on much of campus due to pwer problems
Large portions of the south side of campus are without network connectivity beginning just before 4am, presumably as a result of thunderstorms taking out power. Some buildings which have power do not have network connectivity because the buildings which feed their network do not have power. We have no word from ComEd regarding when power might be restored. |
| 07-14-2008 10:35 PM |
Green |
FindIT Service Restored
FindIT, NSIT's service catalog database, was brought down at approximately 4 PM July 14th by a large-scale power outage in Vancouver, where the service provider is located. Service was restored at approximately 8 PM when the provider was able to transition to emergency power. |
| 07-14-2008 5:23 PM |
Yellow |
FindIT Service Down
FindIT, NSIT's catalog of available services, is currently unavailable. NSIT is contacting the vendor to resolve the outage and restore service as soon as possible. |
| 06-20-2008 1:48 PM |
Green |
South Campus Phone Service Restored
As of approximately 1:00PM all phone services that were impacted by this morning's fiber cable cut have been restored via a re-routing of the infrastructure. |
| 06-20-2008 9:17 AM |
Red |
South Campus Phone Service Outage
11:15AM UPDATE: The fiber is being repaired and for some users service has been restored. However, there remains much to be done and because of the nature of the connections, users may find that their phone remains out while adjacent ones are back in service. We ask staff to please refrain from submitting trouble tickets at this time. Once service has been fully restored, notice will be posted here. Any remaining concerns can then be identified.
10:15AM UPDATE: The fiber cut is still being investigated for repair - no estimate for restoration of full service is available at this time. 4-TECH, 2-9100 and 2-1234 have been rerouted and now have limited capacity for inbound calls.
At approximately 8:20 this morning, portions of the south campus found themselves without University phone services. A fiber communications cable feeding this area was accidentally cut through, curtailing phone service to a number of buildings south of 60th Street. A number of NSIT services, particularly 4-TECH, 2-9100 and 2-1234 University Operators are among those affected. These services are being rerouted in an attempt to answer inquiries arriving about this service outage, but calls may be delayed until services are fully restored.
As further information becomes available, we'll post updates to this notice. |
| 05-20-2008 8:46 AM |
Green |
Systems normal
As of 8:45 AM Tuesday, 20 May, NSIT systems are operating normally. |
| 05-19-2008 3:52 PM |
Yellow |
Slow email performance
On the morning of Monday, 19 May, a rogue process flooded one person's mailbox with tens of thousands of messages in a short period of time. This caused a backup in email delivery and problems logging into email for approximately a quarter of our users.
As of 3:30 PM, the excess messages have been cleared out, but the system is still catching up with the accumulated backlog. As a result, people may still see email problems into the evening.
NSIT is investigating ways of preventing this kind of problem from recurring. |
| 05-19-2008 12:00 PM |
Yellow |
Email Performance
NSIT email is experiencing reduced performance. Accessing email may be very slow (or unavailable). As additional information is available it will be posted here. |
| 05-08-2008 9:29 AM |
Green |
Gargoyle System back up
NSIT has rectified the system issues from this morning.
The application is back up and available as normal
Once again, we appreciate your patience and understanding. |
| 05-08-2008 8:57 AM |
Red |
Gargoyle System Outage
As of around 8am this morning, the Gargoyle system experienced problems and is currently unavailable.
NSIT staff are working diligently to resolve the problem.
Further updates will be provided as the situation becomes clear.
Thank you for your patience and understanding, as we work to stabilize the application |
| 04-25-2008 2:23 PM |
Green |
Gargoyle Performance/Outage Issues
We have isolated the recent performance/outage problems to the running of some queries which produce a report output directly into MS Excel.
It seems that after the query has run on the database, the act of importing records directly into an Excel spreadsheet on the Gargoyle user's system, causes the Webserver to exceed the maximum session memory capacity imposed by the operating system. This is a Microsoft limit and is not reconfigurable.
This has only become apparent recently as reports of a larger nature are being run.
The Gargoyle team has made arrangements with the small subset of users who require these reports, for them to be run externally from Gargoyle.
Consequently, we do not anticipate any further system problems, and if you have not been approached individually, you may assume you can utilize the system as normal.
Once again, the Gargoyle team would like to thank you for your continued patience and understanding as we endeavor to provide a stable and efficient application environment.
|
| 04-24-2008 3:31 PM |
Yellow |
Gargoyle System performance Issues
We are experiencing some intermittent performance issues whereby a particular job causes the system to overload. we have asked the business area that runs this task to refrain, which seems to let the system stay stable.
Gargoyle is generally up and available, but be aware it may encounter random system outages
Further updates will be provided as they develop. |
| 04-24-2008 9:08 AM |
Green |
Windows Login Problems resolved.
A domain login controller in the ADLOCAL Windows domain began refusing connections earlier this morning (Thursday, 24 April 2008). This caused a random subset of administrative users to be unable to log in to their Windows desktop machines. NSIT systems staff have fixed the problem; all login services should now be operating normally. |
| 04-24-2008 8:49 AM |
Red |
Windows Login Problems
The NSIT Support Line is receiving numerous reports of problems logging on to ADLOCAL, the centrally administered Windows domain. Commonly, people are unable to log into their desktop Windows machine, and/or receive an error message about using cached credentials. NSIT systems and support staff are working to isolate and resolve the problem. |
| 04-14-2008 1:35 PM |
Green |
Gargoyle outage update
The current problems on Gargoyle are resolved. Users should be able to perform normal functions. If you encounter any further problems, please send an e-mail to gargoylehelp@listhost.uchicago.edu. System administrators will be monitoring Gargoyle for any further problems. |
| 03-17-2008 10:03 AM |
Yellow |
Password-stealing scam on campus
Over the weekend of March 15th, a large number of people on campus received email purporting to be from "THE UCHICAGO.EDU TEAM". This email asked the recipient to email back with their password to prevent their account from being "deactivated". This message did NOT come from NSIT or any University authority; it was a "phishing scam" used to steal people's passwords and accounts. If you responded to this message, you should change your password immediately. (Network Security is also working to identify accounts stolen by this scam and proactively lock them to prevent misuse.)
This is not the first time this scam has hit the campus, and will not be the last. You should be aware that, by policy, NSIT will NEVER contact you to ask for your password, either by phone or by email. It is also explicitly against University policy to send your password to anyone via email. If you should receive such a request via email, you should delete it without responding. For more information, see the Safecomputing website's article on Phishing at http://safecomputing.uchicago.edu/phishing/. |
