Who Suffered More From the Affair: Hester or Dimmesdale?

When I read the first few chapters of the Scarlet Letter, I was angry with Hester's lover (I didn't know who it was at the time) for just watching her while she stood on the scaffold for her punishment. It made me really mad that Hester was suffering so much- she was completely alienated by her entire community, and he was not. I learned later in the novel that I was completely wrong. It turns out, Dimmesdale was suffering just as much or more than Hester.
Both Hester and Dimmesdale suffered greatly from the affair. In my opinion, the affair affected Dimmesdale much more than it affected Hester. It caused Hester to be ostracized, but Dimmesdale's cowardice in not confessing lead to his death. Hester had a horrible punishment: she had to wear a scarlet letter for the rest of her life. After wearing my scarlet letter, i realize how miserable that must have been for Hester. But I still believe Dimmesdale's internal struggle with his own cowardice and guilt was far worse than a scarlet letter.