Assigning skewed queries to an allocation group that has a very small CPU limit (or ABSolute scheduling policy if you are on an old release) will limit the impact on other work.
I don't think there is a black and white answer on this. I consider the "penalty" AG mainly a way to buy time so you can investigate further.
You have to be careful about canceling queries that appear to be skewed. For one thing, once a problem query starts overconsuming resources on an AMP or AMPs, you often have other queries that "fall behind" on those AMP(s); even after the problem query is out of the system, it may take a while for those others to "catch up" - so if you are looking at PM/API snapshots (e.g. PMON), many queries may appear skewed for some period of time despite having balanced total resource usage. There is also the question of how valuable the answer may be, and whether or not figuring out the issue, fixing the skew, and re-running will really return the answer any faster than just allowing it to finish running. On the other hand, whenever you make a query run longer (by limiting its resource consumption), it ties up resources longer.
Maybe you let the original query run in the "penalty" AG until you have successfully run a "fixed" version - at which point you shouldn't get many complaints about canceling the original one.