In an ideal world you'd have bypasses or tunnels to help them keep up speed, but that all depends on land allocation and investment. If you're stuck with old windy narrow tracks then yes youre going to be limited.

Trains like shinkansen can go very fast through dense urban areas, i think mostly in tunnels - but also because they spend a lot to straighten the routes. Obviously there are still some slow sections, but they minimize it by design - and probably a willingness to bulldoze historical land ownership.

I guess in Philly the route probably winds around a bit. https://www.openrailwaymap.org/ It looks like all lines funnel through a couple of very tight curves around the centre. That looks like quite a sensitive choke point.

But apart from that section it looks like the acela limit is over 100mph for quite a lot of the urban area around Philly, which isn't too bad. Its not like its crawling around at 60mph for half the distance.

There's another interesting looking slow chicane in Wilmington. In that case there's what looks like an ideal bypass line already there for any express. It runs through a massive siding (freight maybe?) that is limited to 10-30mph. Looks like a no brainer to me, strengthen those bridges and run any express through that. plenty of dead space around there to reconfigure just one level crossing i think to worry about. most of it is 3-4 tracks already judging by google map.

I heard that US freight and passenger rail don't like to share and enjoy though so probably that's a non starter.