Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.

Topic: regularization?
Replies: 7   Last Post: Feb 28, 2012 11:16 AM

 Messages: [ Previous | Next ]
 Martin Brown Posts: 229 Registered: 12/13/04
Re: regularization?
Posted: Feb 23, 2012 4:45 AM

On 22/02/2012 20:45, RichD wrote:
> On Feb 16, Martin Brown<|||newspam...@nezumi.demon.co.uk> wrote:
>>> What is regularization, in applied math and stats?
>>
>> In this context some function of the reconstructed
>> model that can be used as a constraint on possible
>> solutions. In effect steering the solution
>> towards desirable properties based on a priori
>> knowledge.

>
> So it's a constraint on the solution space,
> not the domain space?

It is used to encode prior knowledge and this is usually a constraint on
the solution space like positivity or smoothness for example.
>
> It it used exclusively for underdetermined
> systems?

It would be redundant for a system of equations that already had a
unique solution or was over determined to begin with.

>> For instance the sky is everywhere positive is a
>> rather powerful strong
>> constraint in astronomical deconvolution programmes.

>
> ?

Have you ever seen negative sky brightness?

Many simple linear methods of deconvolution like Wiener filters produce
them. They are a strong hint that the reconstruction is physically
impossible. Barrier functions provide a convenient way to enforce the
positivity constraint on the set of possible solutions.

>> This has been encoded in various forms such as
>> maximum entropy sum(log(f)) by Burg or
>> sum(flog(f)) by Gull& Daniell. You can choose
>> various other regularising functions which will
>> introduce different biasses in the final solution.
>> sum(f^2) or sum(|f"|) for instance.

>
> I don't understand what these (f) things are,
> or how they enter the calculations.

f[x,y] is the reconstructed brightness solution as a function of
position (or whatever it is you are trying to determine).
>
>> Gull demonstrated it nicely in his Kangaroos problem. see
>>
>> http://sci.tech-archive.net/Archive
>> /sci.image.processing/2008-10/msg00099.html

>
> It might be a nice demonstration nicely, but
> the article is poorly written.

In what way? Here is a link to Jaynes version:

http://bayes.wustl.edu/etj/articles/cmonkeys.pdf

>> ISTR The original paper is called Monkeys,
>> Kangeroos and N.
>>

>>> Is it synonymous with optimization?
>>> I recently heard a speaker refer to it as
>>> a means of achieving algorithmic
>>> and noise robustness, but I didn't get it.

>>
>> A suitable choice of regularising function can make
>> an otherwise ill posed problem tractable and/or
>> have a unique solution.

>
> It's not clear to me, whether the regularizer
> is applied to the data as a pre-processing step,
> or as an extra set of equations to be satisfied.

It is an extra set of regularising constraints on the problem.

Minimise C = sum[ ((data - model)/sigma)^2)
subject to additional regularising function
Maximise S = sum[ -f.log(f) ]

Look up Lagrange multipliers.

--
Regards,
Martin Brown

Date Subject Author
2/15/12 Rich Delaney
2/16/12 Martin Brown
2/22/12 Rich Delaney
2/22/12 Gordon Sande
2/23/12 Martin Brown
2/23/12 Richard Ulrich
2/28/12 aruzinsky
2/28/12 aruzinsky