The rapid emergence of increasingly powerful Large Language Models (LLMs), especially those augmented with “thinking”-based architectures, has fueled enthusiasm toward specification-driven planning paired with external solvers. However, it remains unclear if these models can reliably encode world dynamics with the semantic accuracy needed for successful plan execution in novel domains. To this end, we introduce ALPSBench, a benchmark for evaluating LLMs' reasoning capabilities in planning model construction. ALPSBench consists of 24 novel PDDL domains that serve as the basis for 2 complementary evaluations: ALPSGEN and ALPSQA, which target PDDL domain generation and diagnostic repair. Our extensive evaluation of 9 state-of-the-art LLMs from 3 different families highlights the significant gap in the reasoning capabilities over such formalisms. Our results show that while modern LLMs achieve high syntactic validity and executable plans in unseen PDDL domains, the plans are rarely validated against their ground-truth counterparts. Additionally, across all models, when given less specific diagnostic prompts for detecting faulty PDDL actions, their performance degrades significantly.