unique factorization of matrices

If I have a set of matrices, call this set U, how can I make this a UFD (unique factorization domain)? In other words, given any matrix $X \in U$, I would be able to factorize X as $X_1 X_2 … X_n$ where $X_i \in U$ and this factorization is unique?

We may assume the matrix entries are real or complex, but I’d prefer not to add additional restrictions on the numbers.

I guess there are many ways to do this, trivially I can take the set $\{pI\}$ where $I$ is the identity matrix and $p$ is a prime number. But I want to have matrices that are more “general”. Thanks!

Edit: There would be a distinguished subset $P \subset U$ which are “primes”. To rephrase the question: what I want is, if I take a bunch of elements $X_i, Y_i \in P$, I can guarantee that $\prod X_i \neq \prod Y_i$, if at least there exists one $X_j \neq Y_j$. Can it be done?

Some of the matrices must be non-commutative, ie $A,B \in P$ and $AB \neq BA$.