The potential reproducibility of amplicon sequencing has been a topic of considerable interest to microbiologists attempting to use this technique to uncover the emergent properties of ecological datasets. In this study, five-way biological and technical replication was employed to assess reproducibility of 16S rRNA amplicon pyrosequencing applied to multiple phases of an experiment designed to determine the impact of micro-droplet encapsulation and ammonium enrichment on a marine microbial community. The experimental design made it possible to investigate how variation in comparative analysis methods affected results. Objective statistical tests (based on Monte Carlo style data permutation) were developed to assess the relative performance of analysis methods that varied with respect to several key factors, and were validated with application to two separate datasets. We find that NGS based amplicon pyrosequencing of 16S rRNA genes is highly reproducible, but that observations can be greatly affected by choices in the analytical approach applied to the data. Objective comparison of multiple analysis methods is necessary to exclude those that may obscure biologically intriguing results and to select those best suited to investigate the hypotheses under investigation.