Lessons Learned

Current State

#

At the end of this migration stage, the following is working is working as intended.

• FortiSwitch management is reachable on VLAN 10.
• Management VLAN is isolated from normal client networks.
• Services VLAN is no longer broadly exposed to normal client networks.
• Pi-hole is online on the Services VLAN.
• Pi-hole SSH path exists on the Management VLAN.
• Pi-hole DNS is being handed out through UDM SE DHCP.
• Normal clients can use Pi-hole DNS.
• Normal clients cannot reach the Pi-hole web admin page.
• APC NMCs received correct management IP assignments after reseating.
• UniFi adoption issues were fixed after allowing inform traffic on TCP 8080.
• Parking VLAN 999 exists for trunk native VLAN hardening.
• All pre-migration devices are tested working.
• All devices have been kicked to their appropriate VLANs (all child devices have been overridden to Kidcave Network, all IoT devices are on the IoT network, etc.)
• Firewall rules have been simplified, upgraded and stress-tested.
• Switchports have been updated and documented/re-labeled.

Lessons Learned

#

Some useful lessons I can extract from this migration:

1. Test VLAN trunking before moving management interfaces. A temporary access port on the new VLAN can prevent lockouts.

2. Same-VLAN access does not prove routed access works as-intended. The FortiSwitch was reachable from VLAN 10 but not from other VLANs until its default gateway was corrected in the management interface.

3. Do not include UniFi inform traffic in generic management-port blocks to avoid UniFi adoption loops. TCP 8080 is required for UniFi adoption/inform and needs to be whitelisted or not included in block rules.

4. Simply unplugging Ethernet does not reboot APC NMCs. The NMC cards had to be physically reseated before they requested new DHCP leases, even after the DHCP server was manually changed on the front panel of both units.

5. Cloud-init can and will conflict with custom Netplan configs. The old 50-cloud-init.yaml file caused the base interface to request DHCP unexpectedly and misbehaved. Even after I renamed the file to 50-cloud-init.yaml.backup, I accidentally left the original file active instead of removing it or using mv.

6. Broad Services access allows scanning, even if firewall rules apear correct. The Services VLAN should use explicit allow rules for approved services, followed by a block rule. It was important to test thje firewall rules before leaving the configuration, as testing revealed that adjustments needed to be made.

7. Gateway traffic is different from routed inter-VLAN traffic. Access to 172.16.x.1 addresses must be handled as gateway/local policy, not just normal internal routing. If set incorrectly, client devices lose access to DHCP and kids become upset when the Internet is down.

8. Understanding the problem is still the most valuable first step to solving any problem. When I took my first Informatics class three years ago, the thing that really stuck was to gather as much information as possible and think through all the different possibilities to ensure the problem is understood well enough that you can solve it effectively. Without pre-planning steps and documentation, changes were sure to be made that were confusing, messy, and potentially unrecoverable. I kept an outline and spreadsheet, alongside screenshots of pre-change settings to ensure I always had a fallback in case a change that was made could be reverted if necessary.

• To add to this, never make large, sweeping changes. As a mold technician at my previous employer, we were taught to make small adjustments and then observe the outcome to ensure we were in control of what was being changed. If you make many changes at once, it can be extremely difficult to understand which change solved the problem or created more work to fix. Making small adjustments saved me a few times from being left with a mess of a migration.

9. Plan to revisit projects later to double-check work, update documentation and respond to unexpected behavior as a result of changes that were made. For example, after the access point rebooted, most devices were able to reconnect without any problems. All services and functions on each device worked as expected. However, the doorbell camera (UniFi G4 Pro) lost the ability to detect and record motion events, even though it continued 24/7 recording. This required a reboot of the camera via the UniFi Protect software and was not detected as a problem until roughly a day after the project was completed.