How is a function of function integrated.eg: sin(cosx)dx?

Answer 1

Not generally possible. Of course, if the derivative of the inner function is there, it's easy, e.g.
Integral of sin x sin(cosx) dx, substitute u = cos x
and then du = - sin x dx
and so the integral becomes the integral of
- sin u du
which is
cos u + C
= cos(cos x) + C
but I'm pretty confident there's no primitive for this example you've given.

Answer 2

Some functions of functions are possible to integrate (such as
Integral ( cos(ln(x) dx), but others aren't.

There will be examples of functions of functions where u-substitution, parts, trig identities, trig substitution, or partial fractions all don't work.

But there is no general method of integrating functions of functions.

Answer 3

You could express sin(cos(x)) as a taylor series and then easily integrate it. This is probably only useful if you want a good numerical approximation of the integral from a to b.