When the Újlaki-Kont clan outgrew the castle of their puszta palace, which played an important role in the times of the Árpád kings, they decided to build a new castle. They chose the area at the junction of the Bakony and the Sárrét, which had been used in Roman times to connect Fehérvár and Veszprém.

Miklós Újlaki started the construction of the castle, which is first mentioned in 1397 under the name of Palota. The fortress played an important role in the battles against the Turks. The great-grandson of the builder, Miklós Újlaky, made significant fortifications, and it became one of the strongest castles in Transdanubia, at which time it was named Újlaki Castle. After the extinction of the Újlaki family, the castle had several owners, and eventually became a royal castle, in which capacity the legendary Turkish hero, György Thury, became its first castle master. Thury was one of the greatest figures of the Hungarian end-time military life, a heroic soldier, a great patriot, a clever commander and a brave soldier. He played an important role in the end-time battles of the conquest, and after the fall of Veszprém and Fehérvár he was the only stronghold to prevent the expansion of the Turks. This was the most glorious period of the palace. In June 1566, Thury with his handful of defenders withstood the attack of the 8,000-strong army of Arslan Pasha of Buda.

The castle, with its regular plan and four corner towers, was rebuilt several times and took its present form. The last major rebuilding was carried out by the Zichy family, who owned the castle. The castle has been continuously built and expanded over the centuries. It was built as a castle, but its walls were reinforced during the end times, the thickness of which can still be seen today. It is a very imposing sight, dominating the city centre and radiating power.

The building is now an event centre, with several beautiful halls for permanent and temporary exhibitions, numerous cultural events and the stylistic characteristics of the fortified castles.