Acknowledgments/apologies (Feb 10). Thanks to those who have taken the time of reading this question and trying to find an answer (I've upvoted the three answers so far). I hope that after some iterations we can get to something good, so please forgive me for changing stuff and for the lack of precision and/or clarity. It's frustrating for me as well, since I'm working for a project where it is expected for us to give priority to performance over (a beautiful) formalization.
Update (Feb 10): I am changing one decent case to indecent, and explaining why. The reason is practical and it comes from the application.
I am working on a problem in which polygons are decomposed in a way similar to triangulation. Before actually decomposing the input, we have to classify the polygon in one of two categories:
- Decent. The polygon is all we ever wanted. We may proceed to the decomposition.
- Indecent. The polygon needs further processing before being decomposed.
After taking a quick look at some books and papers, it seems to me that defining a polygon as a "simple closed curve on the plane" is quite the standard, and also the usual assumption for an input in algorithms (to avoid self-intersections and degenerate cases).
However, we want to make this definition in such a way that simple polygons $\subsetneq$ decent polygons, that is, the class of decent polygons is greater than the class of simple polygons.
Let's forget about simple polygons, then, since they are so nice, and let's try to define "the rest" of decent polygons. We have a rough idea of what we want to define as decent based of examples.
Decent polygons
In the following examples (label them D1 and D2, from left to right), a point on the plane belongs to more than two edges (four edges, in fact), but this is not really a problem since there is no overlap (of a positive area).
The labels of the edges indicate the order in which they are given. This order is important to us because we want to consider orientation of (sub-)polygons to define the interior of the (whole) polygon.
Indecent polygons
Label the following examples I1, I2, I3 and I4, from left to right. This is what we don't want. There is an overlap in the first two examples we'd rather avoid (for example, separating the input in two different inputs, one with edges 1, 2 and 3, and the other with edges 4, 5 and 6). In I3 we have some kind of sub-polygon completely contained in the main polygon. We don't want that either. I will explain later why D2 is not the same as I3.
In I4 there is no overlap, but, as before, we would prefer to have that as two separate inputs, even though we need to introduce an additional vertex. We don't want intersection of segments in points other than their endpoints (i.e., the only intersections allowed are in vertices of the polygon).
Even so, the following polygon I5 is indecent. Note that the triangle 456 is oriented clockwise, and we consider it to have negative area.
Why D1 is not the same as I5
If you follow the path given by the ordered edges, you have that, in D1, the subpath 3-4 (in red) does not cross the subpath 6-1 (in black).
However, in I5, the subpath 3-4 (in red) does cross the subpath 6-1 (in black), and we don't want this.
Why D2 is not the same as I3
In our application, D2 is different from I3. There is no overlap in D2 and its interior can be seen as a shaded area in the following figure.
How to define this?
We're trying to find a definition for these cases. As I said before, "simple" is not enough, but what's more important to us is that the definition should be (if possible) algorithmic, in the sense that an (efficient) algorithm, not a human, should decide whether a given polygon is decent or not. Also, we are willing to relax a case or two if there is a definition that works for almost all cases, but provides an efficient algorithm to verify it.
I don't know whether this has been studied before. Feel free to point me to the appropriate literature if you think it would help, but I have checked a few references and all I've found ranges from "we will assume all polygons are simple" to "non-simple/degenerate cases are left as an exercise".