something like this?
I think in your version it should work, but since you only have one value (distance) instead of 3 (eg. xyz or rgb), you really can't displace in more directions. Use the color rather than the distance and vectormath instead of just regular math node :)
edit:
the voronoi texture doesn't really seem to have more components to it than one. swapped to noise and now the spheres get translated to every direction
