INTRODUCTION
Here are some materials associated with the paper

"Evaluating optimization algorithms: bounds on the
performance of optimizers on unseen problems"
by David Corne and Alan Reynolds

presented at:
GECCO 2011 Workshop on "Scaling Behaviours of Landscapes,
Parameters and Algorithms"

Keep coming back, we will improve what's here over time.

CONTENTS

The files  "upper-m5, "upper-m10", etc, ... contain 
BOUNDS in relation to the performance of an
optimizer on a set of (unseen) problems as follows:


Suppose you test your optimizer on a set of M problems.
(then you want to look at the file:  upper-m<M>)

Now, suppose your optimizer A beats the state of the
art algorithm B on k of these m problems. I.e. it 
has a success rate of k/m.  You can use the tables
to give you a lower bound on the success rate 'in general',
for a given confidence level.

Say that A beats B on 14 problems from a test suite of 
20 problems. And, say you want a lower bound in which
you have 95% confidence.  Now, look at upper-m20
(because there were 20 problems), and look at 
the row for 6 (A lost to B on 6 problems in the test set),
and you find 0.507 -- let's call it 51%.  Now you can
say:

With 95% confidence, A will be beaten by  B in up to 51% of 
problems from the same distribution as the test set.  

Or equivalently:

With 95% confidence, A will beat by  B in at least 49% of 
problems from the same distribution.  


So far the tables give only upper (lower) bounds on the 
failure (success) rate.   We'll add the lower (upper)
bounds asap. 

------------------