↓ Skip to Main Content
Shannon Wetnight

Remediation Summaries

·· loading · loading · ·
Table of Contents

October 2026
#

October 2, 2026
#

Bedroom Switch VLAN Trunk State Failure After Power Cycles

Date/Time of Incident:

  • First detected: 10/02/2026 at approximately 10:30 PM EST
  • Duration: ~1 hour

Affected Devices:

  • UniFi USW Flex 2.5G Bedroom Switch
  • Devices connected through the Bedroom switch uplink

Issue Description
#

After several power cycles, the Bedroom switch remained visible from the FortiSwitch side but appeared offline in UniFi. Devices connected to the switch stopped receiving DHCP leases and showed unidentified network connectivity.

Root Cause
#

The Bedroom switch uplink still showed VLAN 999 as native with VLANs 1-4, 10, and 11 tagged, but the FortiSwitch MAC table showed Site Administrators VLAN 4 clients arriving untagged and being learned on VLAN 999. The running VLAN state on the Bedroom switch appears to have fallen out of sync with its controller configuration after the power cycles.

Remediation Actions
#

  • Verified affected client MAC addresses were being learned on FortiSwitch port 8 under VLAN 999
  • Temporarily changed FortiSwitch port 8 native VLAN from 999 to VLAN 4, which restored DHCP connectivity
  • Re-applied the Bedroom switch uplink tagged VLAN configuration to force a reprovision
  • Returned FortiSwitch port 8 native VLAN to VLAN 999
  • Released and renewed the affected client DHCP lease and confirmed a valid VLAN 4 address

Current Status
#

The Bedroom switch is online again on Management VLAN 10, and connected Site Administrators clients are receiving valid VLAN 4 leases. VLAN 999 remains the native blackhole VLAN on the uplink.

June 2026
#

June 27, 2026
#

APC AP7750 ATS 2 Network Accessibility Failure

Date/Time of Incident:

  • First detected: 06/27/2026 at approximately 10:30 PM EST
  • Duration: ~2 hours

Affected Devices:

  • APC AP7750 ATS (ATS 2, Rack Space 8)

Issue Description
#

APC ATS 2 became inaccessible over the management network and stopped responding to ICMP, HTTP, and other management traffic at its expected IP address (172.16.10.144). Link status remained active on the FortiSwitch and the management card showed network activity, but the device could not be located via standard subnet scans or DHCP lease tables. My initial suspicion pointed toward a physical layer issue or a failed NMC.

Root Cause
#

The ATS NMC was operational at Layer 2 and visible on VLAN 10 with MAC table inspection, ruling out cabling, switchport, and VLAN misconfiguration. Serial console access showed the network stack was configured for DHCP-only and enforcing an APC-specific DHCP vendor cookie requirement. This was previously disabled for both ATS units but somehow re-enabled caused the card to repeatedly reject or stall DHCP lease acceptance, resulting in the device appearing offline despite maintaining network link. I suspect after the lease initially expired, it was unable to renew due to requiring the cookie.

Remediation Actions
#

  • Verified physical connectivity by swapping Ethernet cables and switchports
  • Tested Ethernet cable to make sure it was not detected open/short
  • Confirmed switchport link state and traffic activity on FortiSwitch
  • Performed subnet-wide nmap host discovery to verify the device was absent from the expected IP range
  • Checked FortiSwitch MAC address table to confirm ATS 2 MAC was present on VLAN 10
  • Reviewed UniFi DHCP lease table to confirm ATS 2 was not actively leased
  • Performed NMC soft reset and reseated the management card
  • Established serial console access via USB-to-serial adapter at 2400 baud rate
  • Reviewed TCP/IP configuration and confirmed DHCP-only mode with vendor cookie enforcement, disabled (again)
  • Re-established DHCP lease and restored network communication

Current Status
#

APC ATS 2 is reachable again. Network communication has been restored and management access via web interface and console is functional. Root cause was isolated to DHCP negotiation behavior and (thankfuly) not a hardware failure.

June 21, 2026
#

APC SMT1500RM2U Voltage Anomaly Reports

Date/Time of Incident:

  • First detected: 06/18/2026 at 10:33 PM EST
  • Duration: ~3 Days

Affected Devices:

  • APC SMT1500RM2U (APC 1, Rack Space 1-2)

Issue Description
#

APC 1 began sending repeated email alerts indicating high input voltage compensation (AVR Trim enabled) along with intermittent site wiring fault notifications. Input voltage remained near the upper transfer threshold, causing frequent toggling between passthrough and voltage regulation. It’s worth noting that these email alerts began after a couple of large storms.

Root Cause
#

Input voltage was consistently fluctuating at 127–128V, matching the configured high-transfer threshold of 127V. This caused APC 1 to repeatedly enter AVR Trim mode. Comparison against APC 2 on the same circuit showed stable operation, indicating the issue was isolated to threshold sensitivity and minor sensing variance rather than an upstream electrical fault.

Remediation Actions
#

  • Verified all physical power connections and rack-side equipment connections
  • Confirmed both UPS units were connected to the same branch circuit
  • Compared live voltage and frequency readings between APC 1 and APC 2
  • Ran self-tests on both UPS units (both passed successfully)
  • Increased APC 1 high-transfer threshold from 127V to 130V
  • Monitored input/output voltage behavior after adjustment

Current Status
#

APC 1 stabilized at 126–127V passthrough with no immediate AVR Trim events after threshold adjustment. Minor frequency variance (60–61Hz) remains present on APC 1 and will be monitored for possible long-term calibration drift. I don’t believe an active site wiring fault is present at this time but I haven’t had this checked by a qualified electrician for confirmation.

November 2025
#

November 7, 2025
#

Portainer Agent Update Failures in Docker-Based LXCs (Paperless 107, Nextcloud 108, OctoPrint 109)

Date/Time of Incident:

  • First detected: 11/7/2025 at 7:00 PM EST
  • Duration: ~2 hours

Affected Devices:

  • Proxmox LXCs + Docker Containers – Paperless 107, Nextcloud 108, OctoPrint 109

Issue Description
#

During routine updates of the Portainer agents in multiple Docker-enabled LXCs (IDs 107–109), the following recurring error occurred when attempting to launch the agent container:

error during container init: open sysctl net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission denied: unknown

Issue tracked: https://github.com/opencontainers/runc/issues/4968

Root Cause
#

From maintainer suggestion:

When runc (or Docker/Podman/containerd) runs inside an LXC that has an AppArmor profile, AppArmor misinterprets accesses to /proc as accesses to /sys due to how it resolves paths for detached mounts.
As a result, legitimate writes to /proc/sys/... (like net.ipv4.ip_unprivileged_port_start) are blocked by AppArmor’s /sys deny rules."

Remediation Actions
#

  • Implemented the official AppArmor workaround recommended by upstream maintainers (runc issue #4968):
# Workaround for runc/AppArmor regression in nested LXC environments
lxc.apparmor.profile: unconfined
lxc.mount.entry: /dev/null sys/module/apparmor/parameters/enabled none bind 0 0

Current Status
#

  • Issue has been mitigated, all containers and services are updated and online.
  • GitHub updates will be monitored for adjustments made that revert this workaround, as it is considered less secure by disabling the AppArmour container wrapping. Optional Agent method swap if no remediation is discovered.

November 20, 2025: Maintainer fixed issue in version 6.0.5-2. Issue resolve and AppArmour container wrapping re-established.

September 2025
#

September 1, 2025
#

Unauthorized SNMP Access Attempts to APC1/APC2

Date/Time of Incident:

  • First detected: 10/012025 at 7:07 PM EST
  • Duration: ~32 minutes

Affected Devices:

  • APC1 – Rack Space 1-2 (192.168.1.141)
  • APC2 – Rack Space 3-4 (192.168.1.142)

Issue Description
#

Both APC UPS units reported repeated “Unauthorized user attempting to access the SNMP interface” alerts. Initial source was logged as 192.168.1.160 (Samsung Galaxy Tab S8 Ultra by DHCP assignment), later followed by 192.168.1.155 (WyseNUT NUT server). Alerts correlated with SNMP v1 queries failing authentication.

Root Cause
#

  • Misalignment of SNMP access controls and DHCP assignments:
    • SNMP was configured with default/loose access permissions.
    • WyseNUT server attempted SNMP queries using read/write when only read should have been required.
    • DHCP lease reassignment may have caused temporary source confusion, correlating traffic with the tablet.

Remediation Actions
#

  1. Re-applied existing SNMP access restrictions on both APC NMCs:
    • Allowed managers limited to:
      • WyseNUT (192.168.1.155, static DHCP)
      • Admin workstation (static DHCP).
    • Reconfirmed all unknown or undefined hosts were disabled via APC1/APC2 web UI.
  2. Changed SNMP community access to Read-Only.
  3. Re-applied static DHCP reservations for WyseNUT and admin PC.
  4. Restarted APC Network Management Interfaces, WyseNUT server, and NUT service to apply changes.
  5. Verified no further unauthorized SNMP attempts via APC logs.
  6. Reviewed UDM SE traffic logs for the tablet to confirm no lateral SNMP activity post-update.
  7. Optionally updated tablet’s firmware to latest out of an abundance of caution.

Current Status
#

  • As of 7:52 PM EST on 10/01/2025, no further unauthorized SNMP access alerts observed.
  • SNMP access limited to designated hosts.

Added to Priority Tasks:

  • Migration to SNMPv3 for encrypted/authenticated polling if supported by NUT project.
Reply by Email