Probability is the main tool in the toolkit of a statistician, and provides the foundation for statistical inference. We therefore need to begin by establishing what we mean by probability.
Interestingly and importantly, statisticians disagree on how probability should be defined, and there are (at least) two different interpretations of probability in common use. This filters through to (at least) two different philosophies in statistics: the Frequentists and the Bayesians.
Rather like the Catholics and Protestants of late medieval England, the two schools of thought are in heated disagreement (although without the bloodshed in the case of the statisticians).
Suppose I have a (possibly biased) coin in my hand and I am interested in . How would I go about determining this probability? A natural approach would be to toss the coin a large number of times, say, and count up the number of heads, say. Then I would estimate as
This is the frequency interpretation of probability: I carried out my experiment a large number of times, and counted the proportion of times my event of interest happened. With this interpretation of probability, I require (at least hypothetically) to be able to repeat my experiment in order to assign a probability. Frequentists only allow this specific definition of probability.