Antimatter is not a source of energy for us, it's a method of storing energy, compact but inefficient.
The factor most ignored in discussing interstellar flight is the kinetic energy that must be invested in the ship to make its tons of matter move at a substantial fraction of the speed of light.
It would be a pity if, frustrated by the price of travel, we elected to become a society that never made contact, that never gave SETI a fair chance.
If interstellar travel is as time- or energy- demanding as the above figures indicate, it is far from obvious what the motive for colonization might be.
Present annual world energy consumption is about equal to the annihilation energy of 4 tons of matter.