That would be tied to the table space size. It's under System > Advanced on the Firewall/NAT page.
Do you have any other really large aliases? Or something that loads a bunch of table entries like pfBlocker?
The bogonsv6 list can be quite large when it's loaded, over 54,000 entries, so it will take quite a bit of table space.
When you run out of table room you end up in a chicken-and-egg scenario where it needs to load the rules to increase the table size but it can't increase the table size because it can't load the rules. You can sidestep that by running:
pfctl -F Tables
Then forcing a filter reload (Status > Filter Reload)