That is mostly a myth. They may have worked less than people at the height of the industrial revolution, but even a laborer who was paid a salary had to spend at least several hours per day on average on “not work” things like food preparation, home maintenance, feeding livestock, gathering firewood, repairing and cleaning clothing. Many tasks that are trivial today were highly arduous.
Then to top it all off it was fairly common for the local lord to force them to do extra labor without pay, like maintaining roads or training in a militia.
If you hand wash your clothes and process all your food from scratch you’re most likely the exception.