Doug, I have a hard time to explain why using pointwise Neumann condition is dangerous in a simple term. So I continue to provide indication, and hope one of them will speak to you. If you are familiarize with boundary single layer potential, you will see that the normal derivative have a jump relation related to the local value of the potential (see Colton & Kress book). This show that we can set pointwise Neumann bc at any values, though the Banach L^p norm (1<=p<infinity) of the "trace" can be anything else. There are an infinity harmonic functions that meet the *point-wise* Neumann condition.
I might not fully understand where your Neumann data are from (I understand from measurement), but you have to be careful about how to plug them into the PDE, even if you think you can define it in the boundary.m.