The guide book series for Morocco, Rough Guide calls Sidi Ifni "...the finest and most romantic Art Deco military town every built". By itself, that should be more than good enough reason to visit, even if it is a clear detour.
On the next page, you can visit Place Hassan 2, which is the best, but there are many other nice houses around town as well.
