If you’re interested in mobile Internet and the iPhone (and, really, who isn’t?), the Byteside ByteBlog has posted the results of its “Australian iPhone data test”. The researchers measured 3G download speeds, upload speeds, and ping times, using the iPhone’s Speedtest.net application connected to four major mobile ISPs around Sydney.
We had concurrent access to four iPhone 3GS handsets, one on each of the four Australian networks — Telstra, Optus, Vodafone, and 3. Travelling around the Sydney CBD and Sydney suburban areas, we ran close to 150 individual speed tests. Tests ranged from Manly to Homebush, Annandale to North Sydney, and plenty in between.
Byteside have kindly made the raw data from their study available for download. To analyse the results in more detail, I thought it would be an interesting exercise to try out the Instat statistical analysis package. Instat can be downloaded for free (for non-commercial use) from the Statistical Services Centre.
Importing Byteside’s raw data (available as an Excel file) into Instat was fairly straight forward. It just needed a bit of cleaning up. To start proceedings, Instat’s Summary Tables was used to produce some descriptive statistics of download speeds.
Table 1: Statistical summary of download speeds by ISP
Telstra looks the clear winner in terms of average and maximum download speeds, followed by Optus, Vodafone and Three. However, I don’t think a summary analysis can tell the whole story. I wanted to have a look at the plots of the distributions and run a few tests. Instat obliges with the first request by providing quite a comprehensive graphing capability.
Intstat’s boxplot feature seems to cover all the basics (there’s also scope for plenty of customisation). Below is a comparison of download speeds between the four carriers.
Figure 1: Boxplot of mobile download speeds by ISP
Now let’s have a look at the histograms…
Figure 2: Histogram of mobile download speeds by ISP
|Fig. 2a: Optus
||Fig. 2b: Telstra
|Fig. 2c: Three
||Fig. 2d: Vodafone
Telstra really does own the opposition, but not consistently so. In fact the most frequent download speed for “The Big T” was less than 2Mbps. All four carriers recorded their fair share of quite cruddy speeds, and all four recorded at least one instance where a download attempt didn’t work at all. The reliability of 3G mobile broadband in this country (or Sydney, at least) has some way to go, apparently.
I also ran a couple of statistical tests. As the data aren’t normally distributed, or the ISP download speed variances equal, I chose the non-parametric tests that Instat offers. Firstly, to ask a somewhat redundant question, does network make a difference to download speed?
Sample n Median Ave rank z
1 110 1902.50 305.0 1.81
2 149 2416.00 401.2 10.70
3 149 525.00 163.2 -10.30
4 151 1125.00 257.4 -2.01
H = 167.42 (adjusted for ties) with 3 d.f
Probability > 167.42 = 0
When you ask an obvious question, expect an obvious answer, I suppose. We can comprehensively reject the null hypothesis that all the medians are equal. Yes, download speeds depend on the network that you’re on.
Looking at the histograms above, the Optus and Vodafone distributions share some similarities. Is there a statistically significant difference between these two carriers? Again, the answer looks fairly obvious before the test is even run, but what the hell…
Two-sample test for independent data
Sample n Median Rank sum
optus_d 110 1902.50 16047.0
voda_d 151 1125.00 18144.0
Mann-Whitney U = 9942.00 U’ = 6668.00
Wilcoxon T = 16047.00
On HO: Mean (for U) = 8305.00
Mean (for T) = 14410.00
s.d. = 602.09 (adjusted for tied ranks)
Hence z = 2.72 Significance level is 0.33% for one-sided test
Significance level is 0.66% for two-sided test
0.33% (p=0.0033, one-sided) is highly statistically significant. So we must conclude that the Optus mobile data network does indeed provide faster download speeds on the iPhone than Vodafone in the Sydney areas covered in Byteside’s study. That’s when the download does actually work, of course.
So this post is really more a brief review of the Instat package than mobile network download speeds in Sydney. I really enjoyed using it for the small amount of statistical analysis presented in this post. Instat was capable and quite fun to drive. Although Instat doesn’t have the polish or bells and whistles of a package like SPSS, it does have an intuitive GUI; a useful tool-set; and doesn’t come with the steep learning curve of R. Free for non-commercial use, the price is right too.
Finally, in a conclusion keeping with the overall theme of this post, Telstra is the obvious choice for mobile data carrier in terms of download speeds and reliability. But you’ll certainly pay for the privilege. Optus and Vodafone both represent good value alternatives. Based on the results collected by Byteside, it’s hard to recommend Three. It is also a particular concern that all four networks recorded nil or negligible speeds to a significant proportion of download attempts in the survey.