Let $f$ be such that $f(0)=0$ and $f'(0)=1$. Find iff conditions for normality of successive iterations $f\circ f\circ\ldots$

Let $f$ be an entire function normalized by the conditions $f(0)=0$ and $f'(0)=1$. Find necessary and sufficient conditions on $f$ for normality of the family of successive iterates $\{f\circ f,f\circ f\circ f, f\circ f\circ f\circ f,\ldots \}$.

I know that if $|f'(0)|>1$, it certainly does not. But for $|f'(0)|=1$ I’m having trouble coming with conditions so strong they are necessary and sufficient. Would a condition on the second derivative of $f$ help?

Any help is greatly appreciated. Since I haven’t made much progress on this I’d prefer if I were just given hints, else I feel like I’m asking for answers. Thank you.

Solutions Collecting From Web of "Let $f$ be such that $f(0)=0$ and $f'(0)=1$. Find iff conditions for normality of successive iterations $f\circ f\circ\ldots$"

We have the following relevant fact:

If $\mathscr{F}\subset \mathscr{O}(U)$ is a normal family, then $$\mathscr{F}’ = \{ f’ : f \in \mathscr{F}\}$$ is also a normal family.

The proof is simple, for any sequence $(g_n)$ in $\mathscr{F}’$ select a sequence $(f_n)$ in $\mathscr{F}$ such that $f_n’ = g_n$ for all $n$. By the normality of $\mathscr{F}$, we can extract a compactly convergent subsequence. Without loss of generality – to save subscripting – assume the whole sequence $(f_n)$ is compactly convergent with limit $f$. For every $z_0 \in U$ choose $r > 0$ such that $D_{3r}(z_0)\subset U$. For $z \in D_r(z_0)$ we then have

$$g_n(z) = f_n'(z) = \frac{1}{2\pi i} \int_{\lvert\zeta-z_0\rvert = 2r} \frac{f_n(\zeta)}{(\zeta-z)^2}\,d\zeta.$$

Since the circle $C = \{\zeta : \lvert\zeta-z_0\rvert = 2r\}$ is compact, $f_n$ converges uniformly to $f$ on $C$, and since $\lvert \zeta-z\rvert > r$ for $(\zeta,z) \in C \times D_r(z_0)$, we have

$$\frac{f_n(\zeta)}{(\zeta-z)^2} \to \frac{f(\zeta)}{(\zeta-z)^2}$$

uniformly on $C\times D_r(z_0)$, whence

$$g_n(z) \to g(z) = \frac{1}{2\pi i} \int_{\lvert\zeta-z_0\rvert = 2r} \frac{f(\zeta)}{(\zeta-z)^2}\,d\zeta = f'(z)$$

uniformly on $D_r(z_0)$. Since $z_0$ was arbitrary, it follows that $(g_n)$ converges locally uniformly, hence compactly, on $U$.

This can be iterated, so for a normal family $\mathscr{F}$, the family $\mathscr{F}^{(k)} = \{ f^{(k)} : f \in \mathscr{F}\}$ of $k^{\text{th}}$ derivatives is again normal.

For our setting, where $\mathscr{F} = \{ f^n : n \in \mathbb{N}\}$, we need only look at the derivatives at $0$. If $\mathscr{F}$ is normal, then the sequence $\bigl((f^n)^{(k)}(0)\bigr)_{n\in \mathbb{N}}$ must be bounded for every $k$. Investigating that will lead you to a necessary and sufficient condition.

I think you donʼt have to consider so precisely. The answer is the family ${\frac{d^3 f^n}{dz^3}}$ is normal.