Katherine's increased social and economic status began in autumn of 1372, so that is likely when the affair truly began (although there must have been some attraction prior).
Katherine and Gaunt had four children between 1373 and 1381. The English chronicler Thomas Walsingham wrote of an event in March 1378 when Gaunt and Katherine were seen in public together:
...casting aside every shame of man and the fear of God, allowed himself to be seen riding through the Duchy with his concubine, a certain Katherine Swynford. [Chronicon AngliƦ]
Supposedly Gaunt's family also warned him about the affair's consequences. His legitimate children might well have been concerned about their step-siblings and whatever favors might have been shown to them instead of to the "originals." In 1381, after the Peasants Revolt, Gaunt accepted that his behavior was partially to blame for the public unrest, and broke with Katherine.
After the death of Constance in 1394, however, Gaunt decided to take Katherine as his third wife and Duchess of Lancaster; this would also legitimize his children by her. They were married in Lincoln Cathedral on 13 January 1396. Gaunt had oral permission from Pope Boniface IX, so he wrote to the pope on 1 September 1396, confessing his affair and that it began while he was married but Katherine was not, laying out the details of her importance to the royal family. Boniface wrote a papal bull, declaring the marriage valid. He also legitimized the Beauforts, their four children born before the marriage.
With the papal blessing, everything changed. While Gaunt lived, no one could accuse Katherine of inappropriate behavior. They were free to be a couple in public. The illustration is of a poetic reading at court at the time, and there is speculation that the audience is meant to represent real people and that Katherine is probably one of them. Gaunt only lived another five years, however. Life as Duchess of Lancaster started well, and worked out well for the Beauforts later, but things changed after Gaunt's death. I'll tell you how next time.