After multiple times having problems connecting external DC power supplies, I started to solder the connectors onto pin headers in different ways.
The most stable solution so far is this.
And if it is put in like I did, a little in on the breadboard power rail, the surface of the breadboard help support it.