Autonomous vehicle control is an important subfield of autonomous vehicle research. Many challenges remain to improve the safety and performance of autonomous vehicle control systems in urban driving environments. One such urban driving environment is the roundabout junction, which presents its own unique challenges to potential solutions to autonomous vehicle control. This paper proposes and tests a vehicle control agent as a candidate solution for urban roundabout navigation. The vehicle control agent is based on a hierarchical deep reinforcement learning architecture with a superior network selecting short-term lane-change behaviour and a subordinate network selecting longitudinal acceleration values. The road sequence followed by the agent is selected by a route planner based on Dijkstra’s algorithm. The proposed agent learns to navigate the roundabout environment safely, reaching the goal state in 100% of validation scenarios after training. The agent also outperforms an agent based on the Krauß-following model in 2 out of 5 tested metrics and matches the performance of the Krauß-following model in the remaining 3 metrics.