I think that in the light of the recent controversy about the place possibly known as the Cordoba Center near Ground Zero, the real cause for annoyance on the part of any New Yorker, surely, is why it has taken so long to get going with any serious construction down there. This Wikipedia entry on the Empire State Building, for example, suggests that the building in Midtown was erected in a space of only a few years. That was in the early 1930s – what was so radically different then?
I suspect that if we already had an impressive and dignified piece of architecture in the southern tip of Manhattan, the row about what happens to nearby buildings would not have erupted so much. It seems that planning and political issues are at stake here – after all, places such as Dubai and various parts of Asia put up skyscrapers with great speed these days.
Or maybe the intention all along is that Ground Zero should remain a flat, empty space of land, purely in the form of a place for remembrance.