The problem with this is that each poll is usually conducted in a slightly different manner, which technically cannot be combined into one poll. Polls in themselves contain uncertainty because they are only using a sample of the population. The uncertainty is measurable, which we usually call the margin of error. An inaccurate way to conduct a poll average is to treat each poll as though they were the exact same in uncertainty. The reason for this is that this type of poll average gives less weight to the polls that have less uncertainty.
However, if you account for the variation in polls due to each poll's unique sample size, you can combine the polls into a poll average. This means the only assumption you are making is that each poll was conducted in a similar manner. You can also measure the uncertainty of this poll average.
An example of a poll average that is done using the more accurate method is found here: 2008 Presidential Election poll average.