The answer is $\angle ABC=90^{\circ}.$ Let the angle bisector of $\angle ABC$ meet $AP$ at $K$; the given angle condition implies that $BMKO$ is cyclic, i.e. $AM \cdot AB=AK \cdot AO$ by power of point. If $a, b, c$ are the lengths of the sides of $ABC$, then is easy to see that $BP=2a-c$ and hence by angle bisector theorem $\frac{AK} {AP}=\frac{c} {2a}$. By Menelaus for $\triangle ABP$ and the line $M, O, C$, we obtain $\frac{AO} {AP}=\frac{a^2}{(c-a)^2+a^2}$ and by LoC for triangle $ABP$ we have $AP^2=c^2+(2a-c)^2-2c(2a-c)\cos \beta$. Hence, $$ac=AM \cdot AB=AO \cdot AK=\frac{a^2c}{2a((c-a)^2+a^2)}(c^2+(2a-c)^2-2c(2a-c)\cos \beta),$$which rearranges to $2c(2a-c)\cos \beta=0$, i.e. $\angle ABC=90^{\circ}$.

In fact, it's easier to construct $O'$ first as the center of rotation sending $AM\to BC$, and then $D$ as the image of $B$ under this rotation. This was actually the way I constructed these points during the contest.