You see Anil Castle has powerful units in terms of health, but they are less numerous. During the first week it is a must to have a lot of fodder or bait creatures to sacrifice. Like Sal said the gryphons are pretty much ruled out because of the building order and low chance of getting them. They are also 2-hexed, making them somewhat worse for baiting.
Another issue is the offensive power. The Castle got strong units but due to their low numbers their offense is weaker. Eg 90 Gremlins are far more powerful than 42 Pikemen on the map. Archer's needs the upgrade badly, but if you upgrade them then you lose the chance of Angels+Castle or Angel+Cavalier.
Finally the most important issue. The dwelling fights on the map.
The monk dwelling will severely damage your units because of their already low numbers. The Cavaliers are somewhat more on the easy side but you still need the Angels and it is also annoying to be forced to use your Pikemen as fodder/bait.
The Angel dwell is a nightmare. The native terrain bonus makes them even faster and if you do not have tactics or angels+slow they will slash you hard.
Compare the Castle dwells to say the Tower dwells.
You can take all of Tower's dwells on day 1 with the help of some magic or with ease once Naga's come. The Genies are most annoying but the Nagas and Giants are easier than they seem. The Nagas are slow and you got plenty of bait for them. The Giants are faster but still a walker and are easy to bait and destroy due to their low health.
What is best tactic/way for Castle and Dungeon? I am tend to only this towns. If I start on good terrian, I can will pick tower town, i really dislike movement penalty.
____________