The term itself comes from the Latin cancellarius, which was someone who hung around the cancelli. The cancelli referred to the lattice-work screens that divided the judge and lawyers from the audience. The cancellarii were the clerks who waited by the divider, waiting to be sent as messengers or given other tasks by the officials. At some point, one of these minor administrators was appointed to a more prominent position. The Latin term for the place cancelli became the English words chancery and chancellery to denote the office of a chancellor.
Inspired by the Carolingian administrative system, Edward the Confessor (reigned 1042 - 1066) appointed what some consider the first chancellor in England, a priest named Regenbald. Titled regis cancellarius, he was probably put in charge of the king's clerks and scribes, and his name shows up as witness to charters. Regenbald was given many estates and the status of a bishop, although he was not ordained one.
For centuries the chancellor was a member of the clergy, likely because for centuries in England the clergy was where you could find a literate man. Early chancellors seem to have been chosen because the king wanted someone to deal with the paperwork of charters, and the chancellor was the Keeper of the Great Seal, freeing the king from having to handle the paperwork himself. The picture above is the chancellor's purse, for storing the Seal. (The Keeper of the Great Seal evolved into a separate office.)
Eventually the chancellor became the Chief Justice, managing the law courts. For a time he was the only judge for cases that needed the king's authority. By the time of Edward III, the chancellor's work evolved into a separate tribunal, the High Court of Chancery, and the chancellor could decide cases based on his judgment regardless of the dictates of law. Two of the better known chancellors, who were sometimes referred to as "the keeper of the king's conscience" were under Henry VIII: Sir Thomas More, the only chancellor who ever cleared the day's docket, and Cardinal Thomas Wolsey; after Wolsey's failure to procure an annulment of Henry's marriage to Catherine of Aragon, Henry (and subsequent kings) mostly appointed lay people as chancellor.
The Lord High Chancellor (now just Lord Chancellor) in the Middle Ages was primarily responsible for the functioning of the law courts.
The Chancellor of the Exchequer is responsible for financial matters; the first appointed was Eustace of Fauconberg (c. 1221 - 1228) by Henry III.
After the Archbishop of Canterbury, the chancellor was the highest-ranking non-royal in England.