While there's no doubt that both portable gaming units appeal to different groups and demographics, the bottom line is that both are handheld devices that play video games.
Hence, the comparisons are inevitable, especially when it comes to consumer interest and sales. There's little doubt that Nintendo owned the portable market this generation, as the DS outstripped just about any other gaming platform available by a lot . The 3DS didn't do well out of the gate but has since managed to climb back up the charts. The Vita is mostly unproven; all we have to go by is the approximately 540k it has sold in Japan and its decline since.
Obviously, price remains a determining factor and will likely be the Vita's biggest hurdle. That, and parents around the globe still identify "video games" with "Nintendo," especially when it comes to kids and handheld entertainment. There may be no chance that the Vita can compete with the 3DS on the sales charts. But as Sony always says, it's a marathon and not a sprint and the plain and simple truth is that the Vita can do things the 3DS can't. Furthermore, with all this game support and marketing (Sony seems intent on pushing their new device), this portable could definitely turn some heads and rise quickly up the ranks.
Perhaps the real question is whether or not one can steal market share away from the other. The competition isn't exactly apples and oranges, but…it isn't exactly apples and apples, either. It's a complex situation with two very different companies and two very different situations. Market share in the handheld market is dominated by Nintendo, but let's not forget lifetime sales for the PSP and the fact that the Vita is…well, pretty damn good. That's kind of important to consumers, right?