Hello Lyapunov,
You don't need to know the transconductance values for GM1 and GM2. The error current gets integrated and results in a voltage at the output of the instrumentation amplifier. The key here is that the voltage present at the inputs of both transconductors must be the same.
Here's how it works: forget about HPA for a moment. Look at figure 53 and just assume that the inverting input of GM2 is connected to REFOUT. If there was 1mV present across the inputs of GM1, the voltage on the non-inverting input of GM2 would need to be 1mV above REFOUT. Because this difference appears across R, this voltage is 99 times larger across 99R, thus achieving a total gain of 100. The transfer characteristic of this loop is shown in figure 7. As you can see, you can simply model the inamp+integrator as a first-order low-pass with a gain of 100.
Note: the nice thing about this instrumentation amplifier topology is that it allows you to play with the feedback inputs very much as if they were the signal inputs, but without affecting the input signal. If you wish to see more details on how it works, go to the following article
http://www.analog.com/library/analogdialogue/archives/48-01/bridge_sensors.html
So, now that we have established that you can satisfy the loop using the two inputs independently, we can look at how the HPA works. If you have HPA behave as a simple integrator (again, figure 53), the 1mV from my previous example would appear as 100mV (above REFOUT) at the output of the inamp only to get integrated by HPA. HPA does not like to have its inputs at different voltages, thus swings in the opposite direction. The final result is that the output of the inamp returns to the same potential as REFOUT, and the output of HPA swings to 1mV below REFOUT. This condition satisfies the loop and evidently behaves as a high-pass filter.
The dominant pole of this high-pass filter is located at 100/RC where R and C are the external components used to build the integrator with HPA. This factor of 100 is due to the gain of the inamp, as explained in page 21 of the data sheet. So, now, you have two poles, one at 100/RC and the second at 2kHz, with a gain of 100. And a zero at f=0. That's your transfer function.
But, what happens if you want to add two poles to the feedback to increase low frequency rejection? This is where it gets tricky, because two poles on the feedback will cause too much phase shift and therefore oscillation occurs. Therefore, you need to add a zero to make it stable. This is what the circuit in figure 56 is doing. Rcomp adds such zero. But this is not a Sallen-Key architecture, it is simply an RC low-pass before the integrator (if you ignore Rcomp for a second). Rcomp adds a zero by placing a lower limit to the attenuation of the RC. And the other condition for stability is that this zero can't be more than one decade away from the pole. Figure 58 shows the effect of that zero on the blue trace.
The bottom line is, if you wanted to work out the math for the whole thing, I'd just model the HPA as a lowpass with two poles and a zero and the inamp with a G=100 and a pole at 2kHz, and then use the feedback equation that Harry suggested above.
Regards,
Gustavo