Its legitimately unsafe. Having traffic on one side consistently faster than you and the other side slower greatly reduces the chance of accidents because you can have have some relative assurance that speeding up while merging left is safe and slowing down while merging right is safe. It also keeps faster traffic out of and further away from onramps/offramps.

Drive safe, don't be an asshole, its not that complicated.