ITS Alerts

Subscribe to ITS Alerts feed
The Information Technology Services (ITS) Alerts System is a rapid-delivery method used to distribute timely information on major ITS services, unexpected outages, and maintenance schedules to the Penn State community.
Updated: 57 min 46 sec ago

UCS Scheduled Maintenance

Fri, 09/08/2017 - 10:16
Enterprise IT is planning to update software on the servers that support UCS. This is not a UCS update, it is an update to the operating systems and related code. Users connected to UCS during the update may see an "error unable to connect" message in their mail client while the web interface may become unresponsive for a few minutes. The updates will be performed over three days, 9/12, 9/14 and 9/19 during the normal maintenance window of 5am to 7am. These changes are being tracked as CHG0044016, CHG0044021 and CHG0044022.

Notice of Changes to Data Center Firewall Management (Panorama)

Thu, 09/07/2017 - 17:43
Please see http://alerts.its.psu.edu/alert-4725 for original alert.

CAB approved CHG0043372.

On Tuesday 9/12 , starting at 7 am until Wednesday 9/13 ending at 7 am, a configuration lock must be put on Panorama and the University Park firewalls. If changes within Panorama, the Hershey firewalls or the University Park firewalls are needed, please open an incident against service “Data Center Services” or contact the Operations Center at 865-4662. The operations center will open a ticket on your behalf.

On Tuesday 9/12, starting at 8 pm until Wednesday 9/13 ending at 7 am, the Hershey firewalls will be moved from local management to Panorama management. The remediation process dictates that we must remove a firewall from the HA pair, proceed with the reconfiguration to Panorama management, then bring the firewall back in to service. The process then repeats on the the other firewall. While a firewall is being taken out of and back in to service, a momentary traffic disruption will occur. An outage of up to 5 minutes is possible for some applications depending on their behavior and tolerance for a lost packet in a session. Although the testing process has shown there does not need to be an extended outage, if there is an emergency an outage of up to 60 minutes may be experienced to rebuild the firewalls.

Starting Wednesday 9/13, all Data Center Firewall Management will be done through Panorama. Further information and documentation is still forthcoming. Please see the original post with a summary of the impact to firewall management.

Known Issue impacting TurnItIn

Thu, 09/07/2017 - 08:46
As of 7:07am on September 7, 2017 the TurnItIn.com service is experiencing issues causing problems logging into the TurnItIn site. Penn State IT staff members are monitoring the TurnItIn status page and will update this alert as more information becomes available.

Intermittent Service Disruption Affecting PASS

Wed, 09/06/2017 - 18:44
PASS service administrators noticed a problem that has caused connections to PASS over CIFS or NFS to be slow or disrupted starting around 4:54pm and are investigating.

Resolved: UCS Experienced partial outage

Wed, 09/06/2017 - 14:53
UCS experienced a partial outage from 1:09pm to 1:23pm due to a hardware failure. Enterprise IT have successfully restored UCS services.

WebMail web server upgrades

Wed, 09/06/2017 - 12:16
Enterprise IT is planning to upgrade the web servers that support Penn State WebMail. There should be no outage and users should experience no change in the service. The upgrades will be performed over four days, 9/7, 9/12, 9/14 and 9/19. This change is being tracked as CHG0043913.

Schedule Move of White1 Core Router

Wed, 09/06/2017 - 10:50
At 12:01 AM on Thursday, 9/14/2017

Penn State IT will begin the scheduled relocation of one of the two core routers at PSU During the window
of 12:01 AM to 5:00 AM. We will also be moving parts of the wireless core infrastructure as well. The
majority of network traffic running through this equipment will reroute through alternative equipment and
continue to operate as normal. There may be a partial outage of wireless for roughly 5 to 10 minutes as
active sessions re-establish on the redundant equipment. The exception to the redundancy will be the Research
Network. We will see a significant outage in the Research Network during the entirety of the move. All
services are expected to be fully operational by 5:00 AM.

Reference Service-NOW Change ID: CHG0043795

Resolved: various services affected

Wed, 09/06/2017 - 09:03
A hardware failure affected several services past the maintenance window. Known affected services included DIMC, Imaging, workflow, UCS, and chat. All production systems were returned to full operation by 7:45.

Core Wireless Network Opitmization

Tue, 09/05/2017 - 14:49
During the 5-7AM Maintenance window on September 7th 2017 ENCS will be preforming some optimization work on the core wireless infrastructure. During that time frame, clients on the wireless may experience intermittent connectivity. The disruptions should be brief, however we have reserved the entire maintenance window.

Reference SerivceNOW change request CHG0043957

Research Park UPS Work

Tue, 09/05/2017 - 13:13
On Thursday September 07,2017 an electrical contractor will be working at Research Park (Penn Stater) on the UPS. This work is being completed in the standard 5 AM to 7 AM maintenance window with an expected power outage of approximately 30 minutes. The existing UPS should be able to accomidate this outage, so we do not anticipate any loss of network services. For more information, please reference Service Now Change CHG0043649.

Resolved: Workflow Down

Thu, 08/31/2017 - 08:56
We are currently experiencing a problem with our Enterprise Workflow service. Staff are investigating the problem. We apologize for any inconvenience this may cause.

CPR Software Updates

Thu, 08/31/2017 - 06:42
On Thursday, August 31, 2017 from 05:00 a.m. to 07:00 a.m., EI&O support staff will be deploying an update to CPR, DIMC, ID Provisioning, Account Activation and other supporting services. No outages are expected during this deployment.

Core Wireless Network Opitmization

Wed, 08/30/2017 - 16:24
During the 5-7AM Maintenance window on August 31st 2017 ENCS will be preforming some optimization work on the core wireless infrastructure. During that time frame, clients on the wireless may experience intermittent connectivity. The disruptions should be brief, however we have reserved the entire maintenance window.

Reference ServiceNOW Change CHG0043850

AT&T Wifi outage at Abington Campus

Wed, 08/30/2017 - 15:43
A known error impacting AT&T Wifi at Penn State Abington has been brought to the attention of the IT Service Desk.
Staff are working with the vendor to resolve this issue.

The IT Service Desk will continue to post updates to this alert as more information becomes available.

Thank you,
The IT Service Desk

WebAccess Outage

Wed, 08/30/2017 - 12:12
As of 10:05am, students, faculty and staff members are experiencing issues connecting to websites behind Penn State WebAccess, the University's online authentication system that protects WebMail, LionPATH, Canvas, the Employee Self-Service Information (ESSIC) and more. Penn State IT staff are aware of and are working diligently to rectify the issue. We will post updates as they become available.

Resolved: Known Error impacted Penn State Network (Wired and Wireless)

Tue, 08/29/2017 - 18:34
At 4:40PM, Penn State IT observed a brief, but widespread disruption of network connectivity which included both wired and wireless services. The cause was a core UEN router became unstable. The affected router was restored to operation at 5:10PM

At this time the UEN appears to be stable. Ref ServiceNOW INC0639282.

Yammer Outage

Tue, 08/29/2017 - 18:04
As of approximately 4:23pm Yammer is experiencing an outage. Microsoft is currently investigating the issue, and more information will be shared as it becomes available.

Visit https://portal.office.com/servicestatus for more information.

Notice of Changes to Data Center Firewall Management (Panorama)

Tue, 08/29/2017 - 14:00
(TL,DR) - Sorry, if you manage a Data Center Firewall with Panorama, you need to read and understand this information.

Some time after the initial installation of the Data Center Firewalls, DCS and ENCS became aware from the vendor that the initial management model in use in the Data Center Firewalls is not in line with vendor best practices. The current model where regional (University Park and Hershey) policies and objects are made on a physical device and global policies and objects are made in Panorama is not correct. Vendor best practice is to have all policies and objects within Panorama. The following are benefits of having all policies and objects within Panorama: removing the need to duplicate objects between regions, having one spot to do all policy changes, allowing logging of policies made in Panorama, taking more advantage of device group hierarchies and enabling easier backup and restore procedures in case of hardware failures.

DCS and ENCS, in cooperation with professional services, has been planning and preparing for a migration of local policies to Panorama to align with vendor best practices. We apologize in advance for the inconvenience that implementing this change will cause. Our hope is that the near term inconvenience is far outweighed by long term convenience.

All of the information below is pending approval of CHG0043371 for Hershey and CHG0043372 for University Park.

What does this mean for you?

Change Freezes:
On Friday 9/1, starting at 5 pm until Wednesday 9/6 ending at 7 am, a configuration lock must be put on Panorama and the Hershey firewalls. Local policy and object changes on the University Park firewalls may still be made. If changes within Panorama or the Hershey firewalls are needed, please open an incident against service “Data Center Services” or contact the Operations Center at 865-4662. The operations center will open a ticket on your behalf.

A change freeze will also be implemented for the University Park firewalls at a future date. The date of that change freeze is yet undetermined as we work with professional services to determine the implementation date. This date is planned to be in the third or fourth week of September.

Outages:
On Tuesday 9/5, starting at 8 pm until Wednesday 9/6 ending at 7 am, the Hershey firewalls will be moved from local management to Panorama management. The remediation process dictates that we must remove a firewall from the HA pair, proceed with the reconfiguration to Panorama management, then bring the firewall back in to service. The process then repeats on the the other firewall. While a firewall is being taken out of and back in to service, a momentary traffic disruption will occur. An outage of up to 5 minutes is possible for some applications depending on their behavior and tolerance for a lost packet in a session. Although the testing process has shown there does not need to be an extended outage, if there is an emergency an outage of up to 60 minutes may be experienced to rebuild the firewalls.

At a yet undetermined date, the University Park firewalls will be moved from local management to Panorama management. The remediation process dictates that we must remove a firewall from the HA pair, proceed with the reconfiguration to Panorama management, then bring the firewall back in to service. The process then repeats on the the other firewall. While a firewall is being taken out of and back in to service, a momentary traffic disruption will occur. An outage of up to 5 minutes is possible for some applications depending on their behavior and tolerance for a lost packet in a session. Although the testing process has shown there does not need to be an extended outage, if there is an emergency an outage of up to 60 minutes may be experienced to rebuild the firewalls.

Further information and training regarding the information below will be provided as soon as it is available.

Policy Management Changes:
Starting 9/6 for Hershey firewalls, and a yet undetermined date for University Park firewalls, all firewall policy and object changes will be made within Panorama. Access to the physical firewalls will be significantly reduced to read-only functions necessary for troubleshooting.

Committing policy changes:
-Committing policy becomes a two step process.
1. Commit to Panorama. This makes the running configuration of Panorama aware of the pending changes. There is no change to the policy running on the physical devices.
2. Commit to Device Group. This makes the now committed changes in Panorama be pushed to the physical devices. The policy on the physical devices is now updated per the changes in Panorama.

Changes will also be made to the device group heirarchy such that:
- Each organization will have a "parent device group". In the parent device group policies and objects can be made that will be automatically inherited to child device groups.
- Each virtual firewall (vsys) will be in it's own device group under the parent device group. This allows you to create "global" rules and objects for all vsys in one place, and specific rules and objects for a specific vsys in another place. Objects made in a child device group can be promoted to a parent device group if needed at a later time.
- Firewall policy inheritance will work like this "Shared" Pre Rules (Centrally Managed) -> "Parent Device Group" Pre Rules -> "Child Device Group" Pre Rules -> "Child Device Group" Post Rules -> "Parent Device Group" Post Rules -> "Shared" Post Rules (Centrally Managed) -> Default Rules (Centrally Managed)

Zones and log forwarding profiles:
- Zones will become generic names. Zones will be Inside (formerly ORG-Trust) and Outside (formerly ORG-Untrust). This allows the use of zones at the parent or device group policy level, which is not possible today.
- Log forwarding profiles will also become a generic name, syslog-forwarding. This allows the use of log forwarding profiles at the parent or device group policy level, which is not possible today.

Central Person Registry Degradation

Sat, 08/26/2017 - 07:16
Penn State IT has detected an issue which may result in a timeout to clients attempting to update records in the Central Person Registry. Staff is currently investigating.

Emergency University Enterprise Network Maintenance Saturday Aug 26th

Fri, 08/25/2017 - 18:09
On Saturday August 26th between 5 and 7 AM ENCS will be preforming emergency maintenance on core university routers.

During that time the following services may be impacted:

Penn State Wireless PSU/Eduroam/ATT Wifi. Brief periods of intermittent connectivity

The following off campus sites will have a 5 minute network interruption:

Philly Naval Yard
Forum Place
Shavers Creek

Additionally users of the Research network (Science DMZ) may see intermittent network disruptions.

Reference ServiceNOW change request CHG0043825

Pages