We propose a new task, CitySim, where given a city map and an AV software stack, the simulator can simulate the trip from point A -> B by populating the city around the AV and controlling all aspects of the scene (e.g., vehicles, pedestrians, traffic light states).