The novel opens with Hester being led to the scaffold where she is to be publicly shamed for having committed adultery. Hester is forced to wear the letter "A" on her gown at all times. She has stitched a large scarlet "A" onto her dress with gold thread, giving the letter an air of elegance.
Hester carries Pearl, her daughter, with her. On the scaffold she if asked to reveal the name of Pearl's father, but she refuses. In the crowd Hester recognizes her husband from Amsterdam, Roger Chillingworth.
Chillingworth visits Hester after she is returned to the prison. He tells her that he will find out who the man was, and that he will read the truth on the man's heart. He then forces her to promise never to reveal his true identity.
Hester moves into a cottage bordering the woods. She and Pearl live there in relative solitude.
Hester earns her money by doing stitchwork for local dignitaries, but often spends her time helping the poor and sick. Pearl grows up to be wild, in the sense that she refuses to obey her mother.
Roger Chillingworth earns a reputation as being a good physician. He uses his reputation to get transferred into the same home as Arthur Dimmesdale, an ailing minister. Chillingworth eventually discovers that Dimmesdale is the true father of Pearl, at which point he spends his every moment trying to torment the minister.
One night Dimmesdale is so overcome with shame about hiding his secret that he walks to the scaffold where Hester was publicly humiliated. He stands on the scaffold and imagines the whole town watching him with a letter emblazoned on his chest. While standing there, Hester and Pearl arrive. He asks them to stand with him, which they do. Pearl then...