interesting problem. I found this which might be of help: http://totologic.blogspot.com/2014/01/accurate-point-in-triangle-test.html if you are only using triangles try the barycentric coordinate system to determine if the a point in time is inside or outside the triangle

here's one approach to one aspect of your problem(s):

it only works with triangles though. It may be of help if you can make the line change direction when it goes outside of the boundary adapted from the link I provided earlier

fantastic, so amazing. let me take some time to study it. thank you very much.

here's another way of checking if a point is inside or outside. This works for any polygon, not just triangles:

It's really powerful, thanks very much!