As Jared Bernstein of the Center on Budget and Policy Priorities points out, "Once you get up to the very top of the income scale ... you've got two-thirds of their income coming from nonlabor sources."
(Read more: Meet the new wealthy: The cash hoarders)
That's not to imply that the wealthy are just living off passive income, or that they're not working as hard as the everyday America. Business owners, in fact, often work longer hours than full-time employees.
"When we think about small business owners, these are 24-7 people and they can be working harder than you or me," said Roberton Williams of the Tax Policy Center. He said there is little correlation between the type of income people receive and their level of work.
(Read more: Yes, the rich are different: They don't retire)
Indeed, economist Emmanuel Saez of the University of California at Berkeley writes that since the 1970s, the rising incomes of the wealthy are due largely to a growth in their wages and salary income.
"The evidence suggests that top income earners today are not 'rentiers' deriving their incomes from past wealth but rather the 'working rich."'
Of course, working might also mean "owning."