Short answer: There is meaning behind the nudge amount, but it’s not typically desirable for pixel/point-based canvases, and there isn’t currently a way to get nudging to do what you want, short of configuring the grid as you already noted. Sorry. :(
Note that the term “pixel” is ambiguous: there are both screen pixels and ruler pixels. Personally I always use Points for the ruler units instead of Pixel, largely to avoid such ambiguity, but OmniGraffle’s default ruler scale for Pixel-based rulers is 1 px = 1 pt, so it works out to the same unless you’ve configured a different scale.
OmniGraffle nudges based on screen pixels, because that’s the smallest meaningful nudge amount (especially when using, say, inches for ruler units and the user has zoomed in for finer control). Note that we obey the system scaling factor, so on a retina screen what is (generally) presented as 1 pixel is actually 2 device pixels.
If your canvas uses pixels for the ruler units, then at 100% zoom, a nudge should get you a 1-pixel movement because 1 screen pixel = 1 ruler pixel; but at 400% zoom it should get you a 1/4-pixel movement because 1 screen pixel = 1/4 ruler pixel; etc. (It’s a lot less obvious what’s going on when you’re not at a “nice” zoom level. For instance, at 127% zoom 1 screen pixel = 0.7874 ruler pixels, which is going to look completely non-obvious.
While this is a known limitation, emailing us about this (“Contact Omni” in the Help menu) will help reflect the need for a solution.