There's a difference between stressful jobs and difficult jobs.
Working all day in the field can be a difficult, physically exhausting job. Despite that, it can also be mentally relaxing and emotionally fulfilling. Similarly, a desk job can be mentally challenging and providing a sense of accomplishment, even though it is not physically demanding.
So what are the factors that turn a regular job into a stressful job? Bad organization, bad attitudes, peer pressure, unclear or unrealistic expectations, bad physical and social work environments etc. Lack of vision, tolerance, ability and also the desire for instant or short term gratification can quickly turn the good stress of work into overwhelming bad stress that degrades the quality of life.
Jobs are generally long term endeavors. Stress damages not only individual health, but also long term work performance.
Even the most difficult of jobs do not have to be stressful. As an employee, the most important factors of stress are yours to change - so change them! (As for the things you cannot change, well, either up your abilities and change them anyway or find a new work environment.)