From the start, St. Peter was recognized as the bishop of Rome, and his position was considered a sort of "first among equals." His successor (so far as we know; we cannot always be certain of such early records) was Pope Clement I, who may be the Clement mentioned by Paul in his first letter to the Corinthians. Pope Clement wrote his own letter to the Corinthians, in which he tells them to be unified and heal a schism that was dividing them. It is considered the earliest example of the bishop of Rome acting authoritatively over far-flung Christians. There is no evidence that his right to instruct them was questioned. In fact, the Ethiopian Orthodox Church considers Clement's letter part of the New Testament.
We refer to him as "Pope Clement," but there is no evidence that he used that title himself. The first bishop of Rome to be referred to as pope (from Latin meaning "papa," but you knew that) was Damasus I (c.305 - 384), who grew up benefitting from Constantine's Edict of Milan granting protection to Christians. In fact, since "pope" meant "father," it could be and was used for bishops. Pope Leo I (called "the Great"; 440 - 461) called himself Pope, and from his time forward that title was reserved for the exclusive use of the bishop of Rome.
Rome became a suitable resting place for the bishops of Rome when Constantine granted them the use of the Lateran Palace (see photo), but sometimes the politics in Rome became unstable and prompted the pope to re-locate, not always by choice. In the 1200s the papal court could sometimes be found in Viterbo, or Orvieto or Perugia. When a pope died, the College of Cardinals would meet in his location to hold the Conclave to elect a successor.
Then there was the pope who fled Rome in disguise because of the Holy Roman Emperor. He just happens to be a pope whose time in office—ten and a half years—includes numerous references to items mentioned throughout this blog. I can't wait to tell you more.