There's a few ways around it:
You don't have to click the end of the handle. You can click or drag or box select anywhere along the handle: the dotted line or the end. Note Spine always gives you a mouse over state, so you know what will be clicked before you click.
Shift+drag with no selection does a box select that doesn't snap to whole number frames. Note if you have handles and keys already selected, this will drag them instead. Spacebar or escape or click in empty space deselects. To always get selection, regardless of what you already have selected, you can use the middle mouse button.
You showed a single curve, but in other cases if you are having trouble selecting parts of one curve because you are hitting another curve, hover over the curve you want to select for a moment. Afterward, that curve gets priority over other keys for selection. If you look closely it will be drawn on top of the others after the hover. This is how you can give Spine context for which curve you want to select. If you click a curve it also "focuses" that curve.
