If the machine has a 5mm drill set up @ the machine in the control nesting settings then it will try and drill with that, otherwise it will pocket it with the next possible tool.
We have 5mm drill set up at machine; this is smallest diameter tool. It is drilling 5mm adjustable shelf holes on same job but does not seem to be recognized as an available tool for the 5mm dowel holes created in hardware hole editor for lock dado connectors.
Check your tolerance setting in Control Nesting - it should be at least 0.001" and I leave mine at 0.002" to be safe. It could be a rounding error.
Hope this helps,
jnr
Josh Rayburn
Hall's Edge, Inc.
CNC Machining Service