Machine Learning

Consolidation

2005-09-15T23:02:00.000-07:00

Hi all. I've consolidated my computer science blog, my personal blog, and my website. It can all now be found at http://zigoku.net . Thanks for reading, and see you there.

UAI paper on monotonicity constraints

2005-03-28T18:58:00.000-08:00

Here is the abstract from the paper Tom Dietterich, Angelo Restificar, and myself have just submitted to UAI:

"When training data is sparse, more domain knowledge must be incorporated into the learning algorithm in order to reduce the effective size of the hypothesis space. This paper builds on previous work in which knowledge about qualitative monotonicities was formally represented and incorporated into learning algorithms (e.g., Clark & Matwin's work with the CN2 rule learning algorithm). We show how to interpret knowledge of qualitative influences, and in particular of monotonicities, as constraints on probability distributions, and to incorporate this knowledge into Bayesian network learning algorithms. We show that this yields improved accuracy, particularly with very small training sets."

Full text in pdf or in ps

Why I decided to work for Google

2005-03-26T18:20:00.000-08:00

Cross-post on my other blog.

New research tool

2005-03-25T11:45:00.000-08:00

In case you haven't already, check out http://scholar.google.com/ . It looks pretty nice.

More prior work

2005-03-10T20:14:00.000-08:00

Potharst and Feelders have an article in SIGKDD Explorations, June 2002, about learning monotonic classification trees (with monotonic and non-monotonic, i.e., contradictory to the prior, data). There's no probabilistic stuff in there, but nonetheless it's interesting to see others are pursuing similar routes. Read here.

Qualitative and quantitative priors

2005-03-09T22:54:00.000-08:00

Expert domain knowledge is usually qualitative, not quantitative, and thus, eliciting probability numbers from experts is usually very difficult. One of the core goals of the KI-Learn project is to use qualitative knowledge, and we attempt this with a language whose qualitative statements

are easy and natural to write by domain experts
have well-defined semantics for probability distributions which correspond to experts' intuitions

However, things aren't so simple. Suppose our training data contradicts an expert's qualitative statement. The proper posterior depends on how much we believe in our domain knowledge and how much data we have. In fact, the question is: how strong is our expert's prior? This is where we are forced right back into specifying quantitative aspects of our model (i.e., the numbers that parameterize the expert's prior).

So here is my question: is it ever possible to specify purely qualitative domain knowledge? I suppose the answer is: only if you assume the knowledge is true with probability 1 (which of course is simply making the quantitative part implicit). This is nasty, though. Nobody wants to state something is true with probability 1, but nobody wants to specify probabilities, either. Is there any alternative to picking the lesser of these two evils? It seems the answer is no...?

KIML: future work

2005-03-03T01:46:00.000-08:00

Examples of valuable domain knowledge other than monotonicities include synergistic influence (two things both positively influence an outcome, but their combined effect is greater than mere additivity) and relative strength-of-influence (two things both influence an outcome, but it is known one is a significantly stronger predictor than the other). We have not yet run experiments to test the value of these statements, but we do have defined mappings from such statements to constraints on probability distributions, and we expect similar results as were obtained for monotonicities.

Longer term "knowledge-intensive machine learning" goals at Oregon State are more ambitious: automatic feature engineering, model simplification, etc.

By the way, for those of you interested in details, I can provide a current draft of my thesis (especially if you are willing to provide constructive criticism :-).

KIML: known prior work

2005-03-01T09:50:00.000-08:00

I mentioned Bayes nets as the best existing example of what "knowledge intensive machine learning" is about. There really aren't very many techniques out there for learning given priors or constraints generated from high-level qualitative knowledge, and the few that I can think of are all structural.

Inductive logic programming in some sense fits in here, but the intensional and extensional information must be in the same format -- logical statements -- which is a little unnatural for ML folks, and furthermore, the built systems don't deal well with uncertainty. Probabilistic logic programs (BLPs and SLPs) are a little closer, but they still only deal with structural domain knowledge (first-order rather than propositional). Qualitative reasoning techniques give us natural and intuitive ways to express qualitative domain knowledge (in particular, knowledge other than structural knowledge), but they don't use that knowledge to learn better models from data, and they have only primitive notions of uncertainty (in particular they don't deal with probabilities). Knowledge-based model construction offers ways of using domain knowledge to better answer queries, even ones on probabilities, but they generally assume hand-engineered probabilities, not ones learned from data.

Of course, this leaves out the maxent work which of course is all about finding distributions which satisfy constraints -- clearly something I need to research more.

Knowledge Intensive Machine Learning, Part 1

2005-02-25T19:15:00.000-08:00

Machine learning is all about learning from extensional information (datapoints). The incorporation of intensional information (domain knowledge) is generally done by hand via data encoding and feature and model selection. Doing this well is arguably the most difficult part of a successful machine learning system. It would sure be nice if our learning algorithm could make direct use of qualitative domain knowledge, say for model or feature selection, or reducing the parameter space? This is the idea behind "knowledge intensive machine learning".

Probably the best (though almost trivial) example of this is the simple Bayesian network. A Bayes net is a probabilistic model which can be learned from data, but for which the user can very easily specify high-level qualitative domain knowledge (causality or conditional independence) which significantly constrains the parameter space. What else do we have? Not much.

Here's a motivating example: suppose you are building a simple anti-smoking propaganda Bayes net. You have domain knowledge that tells you what nodes should be connected (for example, smoking and lung cancer). But we know more than that structure: the more a person smokes the more at risk they are for lung cancer. The qualitative reasoning folks would write this as "SmokingFrequency Q+ LungCancer", indicating a qualitative (perhaps stochastic) monotonic influence. Now, instead of using that statement for qualitative simulation, we use it for putting a prior on our CPTs that enforces stochastic monotonicity, resulting in better classifiers in low-data situations.

"Hello World"

2005-02-23T21:13:00.000-08:00

My name is Eric Altendorf. I'm currently studying with Tom Dietterich at Oregon State University (I apologize if that link takes you to a large photograph of cows with glasses; I am absolutely not responsible for the department's graphic design choices). I've been convinced to share some thoughts on machine learning by my friend Yaroslav, though I am quite sure I will have not so much to share as he.