In general, IPSec tunnel setups cannot transfer routing protocols such as OSPF. IPSec does not always support the notion of an interface on which a routing engine (such as ospfd) can rely. (Remember, IPSec deals with SAs.) This can be accomplished by deploying OSPF over IP-IP/GRE tunnels over IPSec or out-of-band routing signaling not taking the crypto path.
Given the caveat mentioned with regard to TTL (TTL=1 breaks tunnel and multicasting) and MTUs, dynamic routing protocols work over tunnels pretty much the same way as over regular point-to-point links, as long as the routing engine can recognize the special interfaces associated with tunnels. Zebra/Quagga can deal with most implementations thanks to sound interface abstraction. For example setups of GRE over IPSec, see http://www.freeswan.ca/docs/HA/HA_VPNS_With_FreeSWAN.html.
The current efforts focus on specification of IKEv2, NAT-Traversal and firewall traversal, opportunistic encryption, DHCP over IKE, tight AES integration, and IPSec domains of interpretation (DOIs) for secure group communication and final touches to the IPv6 architecture. Currently, there is a significant trend toward hardware crypto-accelerator cards and chipsets that relieve the CPU from performing 3DES/AES encryption (and, most recently, that perform hashing calculations in hardware).
AES has improved crypto efficiency greatly; however, sustainable gigabit crypto throughput is still a domain of expensive commercial firewalls. An interesting addition to IPSec is the keynote trust management system that remedies some of the disadvantages of PKIs (introduced in RFC 2704 and RFC 2796). Also visit http://www1.cs.columbia.edu/~angelos/keynote.html.