If I have the function $f(x)$ and I want to apply it $n$ times, what is the notation to use?
For example, would $f(f(x))$ be $f_2(x)$, $f^2(x)$, or anything less cumbersome than $f(f(x))$? This is important especially since I am trying to couple this with a limit toward infinity.
You could define the notation recursively as a sequence of functions.
Let $f_{n+1}(x) = f(f_n(x))$ for $n \geq 2$ with $f_1(x) = f(x)$.
Sequence notation of this type is so generic that the reader will be forced to consult your definition, which will avoid any possible misinterpretation.
In the course I took on bifurcation theory we used the notation $$f^{\circ n}(x).$$
You can use the notation $f^n$ to denote the composition of the function with itself $n$ times, though this may also mean the product of $f$ with itself $n$ times. Just make sure you define your notation at the start.
You can use $f^n(x)$ BUT be sure to tell the reader that you mean functional iteration, not $(f(x))^n$.
If you take function iteration as a fold of self-composition, you can use a sum-like notation:
$\bigcirc^nf = \underbrace{f \circ \dots \circ f}_{n\:\text{times}}$
Where:
$\left({\bigcirc^0f}\right)(x) = x$
Granted, this is not very compact, and I would prefer to typeset the limit directly above the circle. That aside, it does combine tolerably with limit notation:
$\lim_{n\rightarrow\infty}\bigcirc^nf$
There are two common notations for this, in two different communities:
People in quantum physics, functional analysis and similar topics write everything without parentheses and for them, naturally $B^2x=BBx=B(B(x))$.
Some people in albegra treat functions as “modifiers” and when $\alpha$ is a function, then they write the image of $x$ by $\alpha$ as $x\alpha$. For them, $x\alpha^2$ is what I would write $\alpha(\alpha(x))$.
At any case, as others mention, you can use whatever you like, as long as you clarify your notations at the beginning. If you need it a lot, I would go for one of $$f^n(x),\qquad f_n(x),\qquad f^{\circ n}(x),\qquad f^{[n]}(x).$$
You may also use Lagrange’s notation of derivative $\ f^{(n)}(x)\ $ instead of more commonly used notations $\ f^n(x)\ $ or $\ f_n(x)\ $.
EDIT:
Or you can use left indices: $\ ^n f(x)\ $ or $\ _n f(x)\ $ or $\ ^{(n)} f(x)\ $ or $\ _{(n)} f(x)\ $.
There is a notation proposal for iteration of functions. Check if the following can be useful for your purposes: Salov, Valerii. “Notation for Iteration of Functions, Iteral”, arXiv:1207.0152, June 30, 2012, https://arxiv.org/abs/1207.0152; Salov, Valerii. “Inevitable Dottie Number. Iterals of cosine and sine”, arXiv:1212.1027, December 1, 2012, https://arxiv.org/abs/1212.1027.