That's how it's supposed to work, so we're gonna need a lot more information about your environment. Logs would help too.
Linux
A community for everything relating to the GNU/Linux operating system (except the memes!)
Also, check out:
Original icon base courtesy of lewing@isc.tamu.edu and The GIMP
I'll be glad to give more info. I'm not sure where to find the logs to tell you what VLC is doing. See my other comment on the comparison of a browser - I want it to use VLC as if I was browsing websites where it just loads into the existing window.
Try Haruna or Celluloid
So you're trying to only get it to behave one way with one type of file, but different ways with other types of files? Not sure if I'm reading that right.
I'm trying to get any new video I click on to play in the existing instance of VLC after running a first video. Not in a new instance. If VLC is open no other video will ever use that instance. It's like if you load a new webpage in a browser but have to either close the existing browser window first or load into a new tab or window, and I find it difficult to believe that's an accepted behavior.