Yes, but this is an optional part of the base package and one of the places in Moose where you pay for how much you use it. In the end, you can simply comment out the type check if it is causing performance issues.
I would be interested to know what version of Moose you used when you made this judgement? Because, the recent 2.0200 version includes inlined type constraints, which provided a non-trivial boost in performance for the native types as well as custom types (provided you include an inlined version in your custom type). If you were profile your code again, you might find a sizable change (and if not, please feel free to send the profile output to the Moose mailing list for us to review).
I'm summarizing explanations from the #moose irc channel now:
Yes, type checks are the main source for performance issue.
However, they are completely optional. This means you can safely disable them in places where you think you don't need them, and only enable them where you think you do. Keep in mind that writing something manually though won't be faster.
This means, that when performance is an issue, your first step would be to fork your dependency and disable type checks. That should get you almost the same speed as a complete Moose-free rewrite, at a fraction of the cost of an actual rewrite. If that really isn't enough, you're likely to get better performance by rewriting in C, than in Moose-free Perl.