Large Language Models (LLMs) excel in a wide variety of natural language processing (NLP) tasks. However, they struggle with long-horizon planning problems requiring structured reasoning, which is essential for many applications. This has led to increased interest in developing better methods within the Automated Planning (AP) and NLP communities. Amid this surge of interest, a growing number of frameworks and LLM-based methods are being proposed. Yet, it remains unclear what works effectively in practice. The evaluation of these methods is limited by the lack of prompt standardization, testing with the most recent state-of-the-art LLMs, and the absence of large-scale benchmarks. This paper makes contributions by bridging that gap, providing a systematic comparison and insights into natural-language-to-AP frameworks using unified datasets, evaluated through rigorous benchmarking on accuracy, scalability, and robustness, and serving as an investigation to uncover their limitations and strengths. We further introduce GRAN2PLAN, a novel LLM-based framework for constructing PDDL domains. Through this evaluation, we identify common pitfalls and clarify ambiguities in LLM-guided planning formalisms.