The first question you need to ask is, what are you counting? Usually you can find some decent thing to count, instead of just "instructions", to get a good guess. Once you have your big-O value, you can double check the value is right.
int count = 20;
for (int i = 0; i < n; i ++) {
Ok, this obviously runs O(n) times.
if (i == count) {
On a first pass, this is entered O(n/20) times. This may change.
for (int j = 0; j < count; j++) {
This runs O(count) times.
// Some O(1) code
// Maybe a print to console or something
}
count *= 2;
now this gets interesting. Each time count doubles.
So the first inner loop runs after 20 outer loops, and then does 20 O(1) work.
Count doubles. The second inner loop runs after 40 outer loops, and does 40 O(1) work.
Count doubles. The third inner loop runs after 80 outer loops, and does 80 O(1) work.
Count doubles. The forth inner loop runs after 160 outer loops, and does 160 O(1) work in the inner loop.
Now you have to turn this into math. You could guess, then check. But suppose you can't guess?
Well, graph it!
n | inner steps
------+----------------
20 | 20
40 | 20+40 = 60
80 | 20+40+80 = 140
160 | 20+40+80+160 = 300
320 | 20+40+80+160+320 = 620
toss that on a graphing program. What does the graph look like? If it curves, taking the logarithm might give you slope from which you can find the exponential of a polynomial.
Remember to use a scatter plot. You don't care what happens at 60, 100, or 240 all that much. You care about the peaks, which happen at 20 and every doubling. A scatter plot drops your graphed points way down.
Also count the outer loop steps and the comparisons with count to ensure they aren't huge.
Once you have a hypothesis, work out how to prove your answer right.