@stephenw10 After quite a bit more testing, I have narrowed the missing static route problem down to the non-dynamic <gateway_item> shown above. The real puzzler is that rolling back to 23.09.1 (BE right before the upgrade), I only have the two dynamic <gateway_items>. <staticroutes> are the same.
23.09.1 pre-upgrade:
<staticroutes>
<route>
<network>192.168.3.0/24</network>
<gateway>MPLS_ALEX_VTIV4</gateway>
<descr><![CDATA[Alex LAN]]></descr>
</route>
</staticroutes>
<gateways>
<gateway_item>
<interface>opt3</interface>
<gateway>dynamic</gateway>
<name>MPLS_ALEX_VTIV4</name>
<weight>1</weight>
<ipprotocol>inet</ipprotocol>
<descr><![CDATA[Interface MPLS_ALEX_VTIV4 Gateway]]></descr>
<monitor_disable></monitor_disable>
<gw_down_kill_states></gw_down_kill_states>
</gateway_item>
<gateway_item>
<interface>wan</interface>
<gateway>dynamic</gateway>
<name>WAN_DHCP</name>
<weight>1</weight>
<ipprotocol>inet</ipprotocol>
<interval>1000</interval>
<descr><![CDATA[Via Quantum Fiber C5500XK]]></descr>
<gw_down_kill_states></gw_down_kill_states>
</gateway_item>
<defaultgw4>WAN_DHCP</defaultgw4>
<defaultgw6></defaultgw6>
</gateways>
To clean up the errant <gateway_item> required tearing down and rebuilding much of the config:
Delete static route to 192.168.3.0/24 via MPLS_ALEX_VTIV4
Delete MPLS_ALEX_VTIV4 interface assignment
Disable IPsec P1 and P2
Delete gateway MPLS_ALEX_VTIV4
[ Gateway was grayed out (Gateway inactive, interface is missing) before attempting to delete and remains in this state after attempting to delete ]
Delete the IPsec P2
Delete the gateway MPLS_ALEX_VTIV4
[ Gateway is deleted and deleted from config.xml ]
Recreate IPsec P2
Enable IPsec P1 and P2
Add interface ipsec1
Enable interface OPT3 (skip renaming to MPLS_ALEX_VTIV4)
OPT3_VTIV4 gateway is created automatically
Add static route to 192.168.3.0/24 via OPT3_VTIV4
Add the OPT3 rules for site to site traffic and gateway monitoring
Reboot
Did this on both of my systems and they are both rebooting cleanly with the IPsec VTI coming up and passing traffic immediately.
I will add this update to the redmine, but it still does not explain where the non-dynamic <gateway_item> came from, and I'm not sure it addresses the problem that @OhYeah-0 is seeing.
--Larry