First Solution: Let a,b,c be nonnegative real numbers such that x=1+a2,y=1+b2 and z=1+c2. We may assume that c≤a,b, so that the condition of the problem becomes
(1+c2)(1+(1+a2)(1+b2))=(a+b+c)2
The Cauchy-Schwarz inequality yields
(a+b+c)2≤(1+(a+b)2)(c2+1)
Combined with the previous relation, this shows that
(1+a2)(1+b2)≤(a+b)2
which can also be written (ab−1)2≤0. Hence ab=1 and the Cauchy-Schwarz inequality must be an equality, that is, c(a+b)=1. Conversely, if ab=1 and c(a+b)=1, then the relation in the statement of the problem holds, since c=a+b1<b1=a and similarly c<b. Thus the solutions of the problem are
x=1+a2,y=1+a21,z=1+(a2+1a)2
for some a>0, as well as permutations of this. (Note that we can actually assume a≥1 by switching x and y if necessary.)
This problem and solution were suggested by Titu Andreescu.
Second Solution: We maintain the notations in the first solution and again consider the equation
Setting (u,v,w)=(ab,bc,ca), we can write the above equation as
uvw+u2+v2+w2−2u−2v−2w+uvw+2=2(u+v+w)
which is the equality case of the sum of the following three special cases of the AM-GM inequality:
uvw+uvw≥2vw,v2+w2+2vw+1=2(v+w)≥0,u2+1≥2u
Hence we must have the equality cases these AM-GM inequalities; that is, ab=u=1 and a(b+c)=v+w=1. We can then complete our solution as we did in the first solution. This solution was suggested by Zuming Feng.