Infidelity involves not only violating the marraige bed, but also unfaithfulness in a relationship; however, not necessarily sexual activity. Viewing the infidelity in this manner raises the idea that Julia is guilty of infidelity in the family by the manner in which she interacts with her children. This unfaithfulness to her husband is one of the things that Francis finds displeasurable. Another incidence of infidelity is the unfaithfulness to his wife shown by Francis to the social activities she finds so important and valuable; it is this that leads to his hitting her. The hit to Julia is yet another example of the unfaithfulness to the marriage. The most evident incidence of infidelity is Francis's thoughts about Anne. Is thinking the same as doing? Francis is in general a highly conflicted individual. He is involved in a near death accident which causes him to ponder the depth and importance of the life he has created for himself.
It is this questioning of the drab, predictable, and boring life that is the heart of his conflict. He is searching now for something different. He sees the maid and remembers a different time, he ponders an affair with Anne for a change, he expresses his anger to his neighbor on the platform, he uses his power to prevent Clayton from getting a job as a means of retaliation, and finally he hits Julia. Each of these things show his need for a change. He looks to the psychiatrist and finds something to make things different, woodworking. Not that he has simply been cured, but he has found a means of outlet, separate from the social, family, and other pressures of life.
Julia responds to being hit by threatening to leave and then stays, Anne is looking for a "father figure" and sees potential in Francis (quickly mistaken), Clayton is looking for something differnt than the life he sees in Shady Hill, and Francis is looking for a change. I think that this is normal and a perfect example of the manner in which humans behave in general.