I think it's the contrast, not comparison, that Cole Porter (the songwriter) is referring to in linking Granada to Asbury Park.

He gives a number of paired contrasts in the song such as:
-- Earthquake vs. simply a shock
-- Turtle soup vs. Mock-turtle soup
-- "For all time" vs. "simply a lark"

And the key comparison, in the last stanza & title of the song (in reverse order)
"At long last ... [genuine] love" vs. "A fancy, not worth thinking of"

In this context, the contrast is between:
-- Granada: a Spanish renaissance city with world-renowned art, architecture and culture vs.
-- Asbury Park.

You can use your imagination, but with no disrespect to lovers of Asbury Park, he's implying it is the opposite of Grenada, i.e. carnival-barkers, seedy, low-brow, or in contrast to "true love", a "one-night stand" kind of town.