Least Squares and Fourier Analysis

I ended my last post on a somewhat dire note, claiming that least squares can do pretty terribly when fitting data. It turns out that things aren't quite as bad as I thought, but most likely worse than you would expect.
The theme of this post is going to be things you use all the time (or at least, would use all the time if you were an electrical engineer), but probably haven't ever thought deeply about. I'm going to include a combination of mathematical proofs and matlab demonstrations, so there should hopefully be something here for everyone.
My first topic is going to be, as promised, least squares curve fitting. I'll start by talking about situations when it can fail, and also about situations when it is "optimal" in some well-defined sense. To do that, I'll have to use some Fourier analysis, which will present a good opportunity to go over when frequency-domain methods can be very useful, when they can fail, and what you can try to do when they fail.
When Least Squares Fails
To start, I'm going to do a simple matlab experiment. I encourage you to follow along if you have matlab (if you have MIT certificates you can get it for free at http://matlab.mit.edu/).
Let's pretend we have some simple discrete-time process, y(n+1) = a*y(n) + b*u(n), where y is the variable we care about and u is some input signal. We'll pick a = 0.8, b = 1.0 for our purposes, and u is chosen to be a discrete version of a random walk. The code below generates the y signal, then uses least squares to recover a and b. (I recommend taking advantage of cell mode if you're typing this in yourself.)
%% generate data
a = 0.8; b = 1.0;
N = 1000;
ntape = 1:N; y = zeros(N,1); u = zeros(N-1,1);
for n=1:N-2
if rand < 0.02
u(n+1) = 1-u(n);
else
u(n+1) = u(n);
end
end
for n=1:N-1
y(n+1) = a*y(n)+b*u(n);
end
plot(ntape,y);
%% least squares fit (map y(n) and u(n) to y(n+1))
A = [y(1:end-1) u]; b = y(2:end);
params = A;
afit = params(1)
bfit = params(2)
The results are hardly surprising (you get afit = 0.8, bfit = 1.0). For the benefit of those without matlab, here is a plot of y against n:

Now let's add some noise to the signal. The code below generates noise whose size is about 6% of the size of the data (in the sense of L2 norm).
%%
yn = y + 0.1*randn(N,1); % gaussian noise with standard deviation 0.2
A = [yn(1:end-1) u]; b = yn(2:end);
params = A;
afit = params(1)
bfit = params(2)
This time the results are much worse: afit = 0.7748, bfit = 1.1135. You might be tempted to say that this isn't so much worse than we might expect — the accuracy of our parameters is roughly the accuracy of our data. The problem is that, if you keep running the code above (which will generate new noise each time), you will always end up with afit close to 0.77 and bfit close to 1.15. In other words, the parameters are systematically biased by the noise. Also, we should expect our accuracy to increase with more samples, but that isn't the case here. If we change N to 100,000, we get afit = 0.7716, bfit = 1.1298. More samples will decrease the standard deviation of our answer (running the code multiple times will yield increasingly similar results), but not necessarily its correctness.
A more dire way of thinking about this is that increasing the number of samples will increase how "certain" we are of our answer, but it won't change the fact that our answer is wrong. So we will end up being quite certain of an incorrect answer.
Why does this happen? It turns out that when we use least squares, we are making certain assumptions about the structure of our noise, and those assumptions don't hold in the example above. In particular, in a model like the one above, least squares assumes that all noise is process noise, meaning that noise at one step gets propagated to future steps. Such noise might come from a system with unmodelled friction or some external physical disturbance. In contrast, the noise we have is output noise, meaning that the reading of our signal is slightly off. What the above example shows is that a model constructed via least squares will be systematically biased by output noise.
That's the intuition, now let's get into the math.When we do least squares, we are trying to solve some equation Ax=b for x, where A and b are both noisy. So we really have something like A+An and b+bn, where An and bn are the noise on A and b.
Before we continue, I think it's best to stop and think about what we really want. So what is it that we actually want? We observe a bunch of data as input, and some more data as output. We would like a way of predicting, given the input, what the output should be. In this sense, then, the distinction between "input noise" (An) and "output noise" bn is meaningless, as we don't get to see either and all they do is cause b to not be exactly Ax. (If we start with assumptions on what noise "looks like", then distinguishing between different sources of noise turns out to be actually useful. More on that later.)
If the above paragraph isn't satisfying, then we can use the more algebraic explanation that the noise An and bn induces a single random variable on the relationship between observed input and observed output. In fact, if we let A'=A+An then we end up fitting , so we can just define  and have a single noise term.
Now, back to least squares. Least squares tries to minimize , that is, the squared error in the  norm. If we instead have a noisy , then we are trying to minimize , which will happen when  satisfies .
If there actually exists an  such that  (which is what we are positing, subject to some error term), then minimizing  is achieved by setting  to . Note that  is what we would like to recover. So  would be the solution to , and thus we see that an error  introduces a linear error in our estimate of . (To be precise, the error affects our answer for  via the operator .)
Now all this can be seen relatively easily by just using the standard formula for the solution to least squares: . But I find that it is easy to get confused about what exactly the "true" answer is when you are fitting data, so I wanted to go through each step carefully.
At any rate, we have a formula for how the error  affects our estimate of , now I think there are two major questions to answer:
(1) In what way can  systematically bias our estimate for ?
(2) What can we say about the variance on our estimate for ?
To calculate the bias on , we need to calculate , where  stands for expected value. Since  is invertible, this is the same as . In particular, we will get an unbiased estimate exactly when  and  are uncorrelated. Most importantly, when we have noise on our inputs then  and  will (probably) be correlated, and we won't get an unbiased result.
How bad is the bias? Well, if A actually has a noise component (i.e. ), and e is , and we assume that our noise is uncorrelated with the constant matrix , then we get a correlation matrix equal to , which, assuming that  and  are uncorrelated, gives us . The overall bias then comes out to .
I unfortunately don't have as nice of an expression for the variance, although you can of course calculate it in terms of , and .
At any rate, if noise doesn't show up in the input, and the noise that does show up is uncorrelated with the input, then we should end up with no bias. But if either of those things is true, we will end up with bias. When modelling a dynamical system, input noise corresponds to measurement noise (your sensors are imperfect), while output noise corresponds to process noise (the system doesn't behave exactly as expected).
One way we can see how noise being correlated with  can lead to bias is if our "noise" is actually an unmodelled quadratic term. Imagine trying to fit a line to a parabola. You won't actually fit the tangent line to the parabola, instead you'll probably end up fitting something that looks like a secant. However, the exact slope of the line you pick will depend pretty strongly on the distribution of points you sample along the parabola. Depending on what you want the linear model for, this could either be fine (as long as you sample a distribution of points that matches the distribution of situations that you think you'll end up using the model for), or very annoying (if you really wanted the tangent).
If you're actually just dealing with a parabola, then you can still get the tangent by sampling symmetrically about the point you care about, but once you get to a cubic this is no longer the case.
As a final note, one reasonable way (although I'm not convinced it's the best, or even a particularly robust way) of determining if a linear fit of your data is likely to return something meaningful is to look at the condition number of your matrix, which can be computed in matlab using the cond function and can also be realized as the square root of the ratio of the largest to the smallest eigenvalue of . Note that the condition number says nothing about whether your data has a reasonable linear fit (it can't, since it doesn't take  into account). Rather, it is a measure of how well-defined the coefficients of such a fit would be. In particular, it will be large if your data is close to lying on a lower-dimensional subspace (which can end up really screwing up your fit). In this case, you either need to collect better data or figure out why your data lies on a lower-dimensional subspace (it could be that there is some additional structure to your system that you didn't think about; see point (3) below about a system that is heavily damped).
I originally wanted to write down a lot more about specific ways that noise can come into the picture, but I haven't worked it all out myself, and it's probably too ambitious a project for a single blog post anyways. So instead I'm going to leave you with a bunch of things to think about. I know the answers to some of these, for others I have ideas, and for others I'm still trying to work out a good answer.
1) Can anything be done to deal with measurement noise? In particular, can anything be done to deal with the sort of noise that comes from encoders (i.e., a discretization of the signal)?
2) Is there a good way of measuring when noise will be problematic to our fit?
3) How can we fit models to systems that evolve on multiple time-scales? For example, an extremely damped system such as

where M >> c. You could take, for example, , , in which case the system  behaves almost identically to the system

with  set to the derivative of . Then the data will all lie almost on a line, which can end up screwing up your fit. So in what exact ways can your fit get screwed up, and what can be done to deal with it? (This is essentially the problem that I'm working on right now.)
4) Is there a way to defend against non-linearities in a system messing up our fit? Can we figure out when these non-linearities occur, and to what extent?
5) What problems might arise when we try to fit a system that is unstable or only slightly stable, and what is a good strategy for modelling such a system?
When Least Squares Works
Now that I've convinced you that least squares can run into problems, let's talk about when it can do well.
As Paul Christiano pointed out to me, when you have some system where you can actually give it inputs and measure the outputs, least squares is likely to do a fairly good job. This is because you can (in principle) draw the data you use to fit your model from the same distribution as you expect to encounter when the model is used in practice. However, you will still run into the problem that failure to measure the input accurately introduces biases. And no, these biases can't be eradicated completely by averaging the result across many samples, because the bias is always a negative definite matrix applied to  (the parameters we are trying to find), and any convex combination of negative definite matrices will remain negative definite.
Intuitively, what this says is that if you can't trust your input, then you shouldn't rely on it strongly as a predictor. Unfortunately, the only way that a linear model knows how to trust something less is by making the coefficient on that quantity "smaller" in some sense (in the negative definite sense here). So really the issue is that least squares is too "dumb" to deal with the issue of measurement error on the input.
But I said that I'd give examples of when least squares works, and here I am telling you more about why it fails. One powerful and unexpected aspect of least squares is that it can fit a wide variety of non-linear models. For example, if we have a system , then we just form a matrix  and , where for example  is actually a column vector where the th row is the cosine of the th piece of input data. This will often be the case in physical systems, and I think is always the case for systems solved via Newton's laws (although you might have to consolide parameters, for example fitting both  and  in the case of a pendulum). This isn't necessarily the case for reduced models of complicated systems, for example the sort of models used for fluid dynamics. However, I think that the fact that linear fitting techniques can be applied to such a rich class of systems is quite amazing.
There is also a place where least squares not only works but is in some sense optimal: detecting the frequency response of a system. Actually, it is only optimal in certain situations, but even outside of those situations it has many advantages over a standard discrete Fourier transform. To get into the applications of least squares here, I'm going to have to take a detour into Fourier analysis.
Fourier Analysis
If you already know Fourier analysis, you can probably skip most of this section (although I recommend reading the last two paragraphs).
Suppose that we have a sequence of  signals at equally spaced points in time. Call this sequence , , , . We can think of this as a function , or, more accurately, . For reasons that will become apparent later, we will actually think of this as a function .
This function is part of the vector space of all functions from  to . One can show that the functions on  defined by

with  ranging from  to , are all orthogonal to each other, and thus form a basis for the space of all functions from  to  (now it is important to use  since the  take on complex values). It follows that our function  can be written uniquely in the form , where the  are constants. Now because of this we can associate with each  a function  given by .
An intuitive way of thinking about this is that any function can be uniquely decomposed as a superposition of complex exponential functions at different frequencies. The function  is a measure of the component of  at each of these frequencies. We refer to  as the Fourier transform of .
While there's a lot more that could be said on this, and I'm tempted to re-summarize all of the major results in Fourier analysis, I'm going to refrain from doing so because there are plenty of texts on it and you can probably get the relevant information (such as how to compute the Fourier coefficients, the inverse Fourier transform, etc.) from those. In fact, you could start by checking out Wikipedia's article. It is also worth noting that the Fourier transform can be computed in  time using any one of many "fast Fourier transform" algorithms (fft in matlab).
I will, however, draw your attention to the fact that if we start with information about  at times , then we end up with frequency information at the frequencies . Also, you should really think of the frequencies as wrapping around cyclically (frequencies that differ from each other by a multiple of  are indistinguishable on the interval we sampled over), and also if  is real-valued then , where the bar means complex conjugate and  is, as just noted, the same as .
A final note before continuing is that we could have decomposed  into a set of almost any  frequencies (as long as they were still linearly independent), although we can't necessarily do so in  time. We will focus on the set of frequencies obtained by a Fourier transform for now.
When Fourier Analysis Fails
The goal of taking a Fourier transform is generally to decompose a signal into component frequencies, under the assumption that the signal itself was generated by some "true" superposition of frequencies. This "true" superposition would best be defined as the frequency spectrum we would get if we had an infinitely long continuous tape of noise-free measurements and then took the continuous Fourier transform.
I've already indicated one case in which Fourier analysis can fail, and this is given by the fact that the Fourier transform can't distinguish between frequencies that are separated from each other by multiples of . In fact, what happens in general is that you run into problems when your signal contains frequencies that move faster than your sampling rate. The rule of thumb is that your signal should contain no significant frequency content above the Nyquist rate, which is half the sampling frequency. One way to think of this is that the "larger" half of our frequencies (i.e.  up through ) are really just the negatives of the smaller half of our frequencies, and so we can measure frequencies up to roughly  before different frequencies start to run into each other.
The general phenomenon that goes on here is known as aliasing, and is the same sort of effect as what happens when you spin a bicycle wheel really fast and it appears to be moving backwards instead of forwards. The issue is that your eye only samples at a given rate and so rotations at speeds faster than that appear the same to you as backwards motion. See also this image from Wikipedia and the section in the aliasing article about sampling sinusoidal functions.
The take-away message here is that you need to sample fast enough to capture all of the actual motion in your data, and the way you solve aliasing issues is by increasing the sample rate.
A trickier problem is the "windowing" problem, also known as spectral leakage. [Note: I really recommend reading the linked wikipedia article at some point, as it is a very well-written and insightful treatment of this issue.] The problem can be summarized intuitively as follows: nearby frequencies will "bleed into" each other, and the easiest way to reduce this phenomenon is to increase your sample time. Another intuitive statement to this effect is that the extent to which you can distinguish between two nearby frequencies is roughly proportional to the number of full periods that you observe of their difference frequency. I will make both of these statements precise below. First, though, let me convince you that spectral leakage is relevant by showing you what the Fourier transform of a periodic signal looks like when the period doesn't fit into the sampling window. The first image below is a plot of y=cos(t), and the second is a snapshot of part of the Fourier transform (blue is real part, green is imaginary part). Note that the plot linearly interpolates between sample points. Also note that the sampling frequency was 100Hz, although that is almost completely irrelevant.

The actual frequency content should be a single spike at , so windowing can in fact cause non-trivial issues with your data.
Now let's get down to the actual analytical reason for the windowing / spectral leakage issue. Recall the formula for the Fourier transform: . Now suppose that  is a complex exponential with some frequency , i.e. . Then some algebra will yield the formula
,
which tells us the extent to which a signal at a frequency of  will incorrectly contribute to the estimate of the frequency content at . The main thing to note here is that larger values of  will cause this function to become more concentrated horizontally, which means that, in general (although not necessarily at a given point), it will become smaller. At the same time, if you change the sampling rate without changing the total sampling time then you won't significantly affect the function. This means that the easiest way to decrease windowing is to increase the amount of time that you sample your signal, but that sampling more often will not help you at all.
Another point is that spectral leakage is generically roughly proportional to the inverse of the distance between the two frequencies (although it goes to zero when the difference in frequencies is close to a multiple of ), which quantifies the earlier statement about the extent to which two frequencies can be separated from each other.
Some other issues to keep in mind: the Fourier transform won't do a good job with quasi-periodic data (data that is roughly periodic with a slowly-moving phase shift), and there is also no guarantee that your data will have good structure in the frequency domain. It just happens that this is in theory the case for analytic systems with a periodic excitation (see note (1) in the last section of this post — "Answers to Selected Exercises" — for a more detailed explanation).
When Fourier Analysis Succeeds
Despite issues with aliasing and spectral leakage, there are some strong points to the Fourier transform. The first is that, since the Fourier transform is an orthogonal map, it does not amplify noise. More precisely, , so two signals that are close together have Fourier transforms that are also close together. This may be somewhat surprising since normally when one fits  parameters to a signal of length , there are significant issues with overfitting that can cause noise to be amplified substantially.
However, while the Fourier transform does not amplify noise, it can concentrate noise. In particular, if the noise has some sort of quasi-periodic structure then it will be concentrated over a fairly small range of frequencies.
Note, though, that the L2 norm of the noise in the frequency domain will be roughly constant relative to the number of samples. This is because, if  is the "true" signal and  is the measured signal, then , so that . Now also note that the number of frequency measurements we get out of the Fourier transform within a fixed band is proportional to the sampling time, that is, it is . If we put these assumptions together, and also assume that the noise is quasi-periodic such that it will be concentrated over a fixed set of frequencies, then we get  noise distributed in the L2 sense over  frequencies, which implies that the level of noise at a given frequency should be . In other words, sampling for a longer time will increase our resolution on frequency measurements, which means that the noise at a given frequency will decrease as the square-root of the sampling time, which is nice.
My second point is merely that there is no spectral leakage between frequencies that differ by multiples of , so in the special case when all significant frequency content of the signal occurs at frequencies that are multiples of  and that are less than  all problems with windowing and aliasing go away and we do actually get a perfect measure of the frequency content of the original signal.
Least Squares as a Substitute
The Fourier transform gives us information about the frequency content at . However, this set of frequencies is somewhat arbitrary and might not match up well to the "important" frequencies in the data. If we have extra information about the specific set of frequencies we should be caring about, then a good substitute for Fourier analysis is to do least squares fitting to the signal as a superposition of the frequencies you care about.
In the special case that the frequencies you care about are a subset of the frequencies provided by the Fourier transform, you will get identical results (this has to do with the fact that complex exponentials at these frequencies are all orthogonal to each other).
In the special case that you exactly identify which frequencies occur in the signal, you eliminate the spectral leakage problem entirely (it still occurs in theory, but not between any of the frequencies that actually occur). A good way to do this in the case of a dynamical system is to excite the system at a fixed frequency so that you know to look for that frequency plus small harmonics of that frequency in the output.
In typical cases least squares will be fairly resistant to noise unless that noise has non-trivial spectral content at frequencies near those being fit. This is almost tautologically true, as it just says that spectral leakage is small between frequencies that aren't close together. However, this isn't exactly true, as fitting non-orthogonal frequencies changes the sort of spectral leakage that you get, and picking a "bad" set of frequencies (usually meaning large condition number) can cause lots of spectral leakage even between far apart frequencies, or else drastically exacerbate the effects of noise.
This leads to one reason not to use least squares and to use the Fourier transform instead (other than the fact that the Fourier transform is more efficient in an algorithmic sense at getting data about large sets of frequencies —  instead of ). The Fourier transform always has a condition number of , whereas least squares will in general have a condition number greater than , and poor choices of frequencies can lead to very large condition numbers. I typically run into this problem when I attempt to gain lots of resolution on a fixed range of frequencies.
This makes sense, because there are information-theoretic limits on the amount of frequency data I can get out of a given amount of time-domain data, and if I could zoom in on a given frequency individually, then I could just do that for all frequencies one-by-one and break the information theory bounds. To beat these bounds you will have to at least implicitly make additional assumptions about the structure of the data. However, I think you can probably get pretty good results without making too strong of assumptions, but I unfortunately don't personally know how to do that yet.
So to summarize, the Fourier transform is nice because it is orthogonal and can be computed quickly. Least squares is nice because it allows you to pick which frequencies you want and so gives you a way to encode additional information you might have about the structure of the signal.
Some interesting questions to ask:
(1) What does spectral leakage look like for non-orthogonal sets of frequencies? What do the "bad" cases look like?
(2) What is a good set of assumptions to make that helps us get better frequency information? (The weaker the assumption and the more leverage you get out of it, the better it is.)
(3) Perhaps we could try something like: "pick the smallest set of frequencies that gives us a good fit to the data". How could we actually implement this in practice, and would it have any shortcomings? How good would it be at pulling weak signals out of noisy data?
(4) What in general is a good strategy for pulling a weak signal out of noisy data?
(5) What is a good way of dealing with quasi-periodic noise?
(6) Is there a way to deal with windowing issues, perhaps by making statistical assumptions about the data that allows us to "sample" from possible hypothetical continuations of the signal to later points in time?
Take-away lessons
To summarize, I would say the following:
Least squares

good when you get to sample from a distribution of inputs that matches the actual distribution that you're going to deal with in practice
bad due to systematic biases when noise is correlated with signal (usually occurs with "output noise" in the case of a dynamical system)

Fourier transform

good for getting a large set of frequency data
good because of small condition number
can fail due to aliasing
also can be bad due to spectral leakage, which can be dealt with by using least squares if you have good information about which frequencies are important

Answers to selected exercises
Okay well mainly I just feel like some of the questions that I gave as exercises are important enough that you should know the answer. There isn't necessarily a single answer, but I'll at least give you a good way of doing something if I know of one. I've added a fold so you can avoid spoilers. It turns out that for this post I only have one good answer, which is about dealing with non-linear dynamical systems.

We can figure out if a dynamical system is non-linear (and get some quantitative data about the non-linearities we're dealing with) by inputting a signal that has only a few frequencies (i.e., the superposition of a small number of sines and cosines) and then looking at the Fourier transform of the response. If the system is completely linear, then the response should contain the same set of frequencies as the input (plus a bit of noise). If the system is non-linear but still analytic then you will also see responses at integer linear combinations of the input frequencies. If the system is non-analytic (for example due to Coulombic friction, the type of friction you usually assume in introductory physics classes) then you might see a weirder frequency response.

Least Squares and Fourier Analysis


Linear Control Theory: Part I

Last time I talked about linear control, I presented a Linear     Quadratic Regulator as a general purpose hammer for solving linear     control problems. In this post I'm going to explain why LQR by itself is     not enough (even for nominally linear systems). (Author's note: I got to the end of the post and realized I didn't fulfill my promise in the previous sentence. So it's redacted, but will hopefully be dealt with in a later post.) Then I'm going to do    my  best to introduce a lot of the standard ideas in linear control    theory.
My motivation for this is that, even though these ideas have a     reasonably nice theory from a mathematical standpoint, they are     generally presented from an engineering standpoint. And although all of     the math is right there, and I'm sure that professional control     theorists understand it much better than I do, I found that I had to go     to a lot of effort to synthesize a good mathematical explanation of   the   underlying theory.
However, this effort was not due to any inherent difficulties in the     theory itself, but rather, like I said, a disconnect in the intuition     of, and issues relevant to, an engineer versus a mathematician. I'm   not   going to claim that one way of thinking is better than the other,   but  my  way of thinking, and I assume that of most of my audience,   falls  more  in line with the mathematical viewpoint. What's even better   is  that many  of the techniques built up for control theory have    interesting ramifications when considered as statements about vector    spaces. I hope that you'll find the exposition illuminating.
As before, we  will consider a linear system

where  and  are matrices and  is a vector   of control inputs ( is the state of the system). However, in   addition to a control input , we will have an output ,   such that  is a function of  and :

In some cases,  will be a set of observed states of a   system, but in principal  can be any quantity we care about,   provided that it is a linear function of state and control. We further   assume that , , , and  are constant   with respect to time. We call a system that follows this assumption a linear   time-invariant system, or just LTI system.
Since the system is linear, we have superposition and therefore can   break up any function (for example, the function from  to   ) into a function from each coordinate of  to   each coordinate of . For each of these functions, we can   take their Laplace transform. So, we start with

and end up  with (after taking the Laplace transform)

Solving these two equations for  as a function of    gives . We call this mapping from  to  the transfer function of the system. Cramer's Rule implies that the transfer function of any linear time-invariant system   will be a matrix where each entry is a ratio of two polynomials. We   refer to such transfer functions as rational. I will show later   that the converse is also true: any rational matrix is the transfer   function of some LTI system. We call such an LTI system the state-space   representation of the transfer function. (I apologize for throwing   all this terminology at you, but it is used pretty unapologetically in   control systems literature so I'd feel bad leaving it out.)
As an example, consider a damped harmonic oscillator with an external   force  as a control input, and suppose that the outputs we   care about are position and velocity. We will let  denote the   position of the oscillator. This has the following state-space   representation:

Here  is the spring constant of the oscillator and    is the damping factor. For convenience we will write  instead   of   and   instead of . Also, we will let  denote the  identity matrix. Then, after taking the Laplace transform, we   get

Solving the first equation gives

or

Therefore, the transfer function from  to  is   .
We can think of the transfer function as a multiplier on the   frequency spectrum of  (note that  is allowed to be an   arbitrary complex number; if  is non-real then we have   oscillation at a frequency equal to the imaginary part of ; if    then we have damped oscillation, whereas if   then the magnitude of the oscillation increases   exponentially. Note that  denotes the real part of .
Exercise: What does a pole of a transfer function correspond to? What   about a zero? Answers below the fold.
If a transfer  function has a pole, then it means that even if a  given frequency  doesn't show up in the input , it can still  show up in the  output . Thus it is some self-sustaining,  natural mode of the  system. For LTI systems, this corresponds to an  eigenvector of the  matrix , and the location of the pole is  the corresponding  eigenvalue.
A zero, on the other hand, means that a mode will not show up in the   output even if it is present in the input. So for instance, the damped   oscillator has poles at . Let us   assume that  and  are both positive for the damped   oscillator. Then, for , both of the poles are   real and negative, meaning that the system is critically damped. For   , the poles have negative real part and   imaginary part equal to , meaning that the   system will exhibit damped oscillation. Finally, there is a zero in  the  second coordinate of the transfer matrix at . This   corresponds to the fact that a harmonic oscillator can be held at a   fixed distance from its natural fixed point by a fixed external force.   Since the distance is fixed, the contribution to velocity is zero.
There is more to be said on transfer functions, but before I go into   that I would like to give you a working picture of how  and    should be viewed mathematically. This is a view that I only   recently acquired. For this I owe thanks to Stefano Stramigioli, who   gave a very   interesting talk on port-Hamiltonian methods at Dynamic Walking   2010. (Update: Stefano recommends this book as a resource for learning more.)
Duality
Here is how I think you should think about linear control   mathematically. First, you have a state-space . You also have a   space of controls  and a space of outputs . Finally,   you have a space , the tangent space to .
Ignoring  and  for a moment, let's just focus on    and . We can think of elements of  as   generalized forces, and the elements of  as generalized   velocities. I realize that state-space also takes position into account,   but you will note that no external forces show up in the equations for   position, so I think this view still makes sense.
If we have a set of forces and velocities, then we can compute power   (if our system is in regular Cartesian coordinates, then this is just   ). In this way, we can think of    and  as dual to each other. I think that generalized   velocities are actually somehow supposed to live in the cotangent space   , rather than , but I don't know enough analysis  to  see why this is true. If someone else does, I would love to hear  your  explanation.
At any rate, we have these two spaces,  and , that   are in duality with each other. The operator  then   induces a map  from  to , where  is   the space of Lesbegue-integrable   functions from  to  (although in practice all of   our inputs and outputs will be real-valued, not complex-valued, since   the systems we care about are all physical). Since  and  are in duality with each other, we can also think of this as   assigning a power history to any force history (the power history being   , where  is the force history).
What's  more remarkable is that the transfer function from force  histories to  state histories is  in the Laplace  domain (as  discussed above — just set  for the  state-space  representation). Therefore it is invertible except on a set  of measure   zero (the poles of ) and so as far as  spaces are concerned it is an isomorphism; this is a   bit of a technical point here, but I'm using the fact that  spaces are composed of equivalence classes of functions   that differ on sets of measure zero, and also probably implicitly using   some theorems from Fourier analysis about how the Fourier (Laplace)   transform is an isomorphism from  to   itself. I'm still glossing over some technical details here; in   particular, I think you might need to consider the intersection of    and  instead of just , and   also the target space of the Fourier transform is really   , not   , but these details aren't really important   to the exposition.
Getting back on track, we've just shown that  the dynamics matrix   of a linear system induces an isomorphism  between force  histories and state histories. My guess is that you can  also show this  for reasonably nice non-linear systems, but I don't have a  proof off  the top of my head. So, letting  denote the space of  control  signals and  the space of outputs, what we have is  something  like this:

Incidentally,  that middle map (the isomorphism with ) is  hideous-looking, and if someone has a good way to typeset such a thing I  would like to know about it.
In any case,  in this context it is pretty easy to see how the inputs and outputs play  dual roles to each other, and in fact if we replaced , , and  each with their adjoints , , and , then we get a new dynamical  system where the inputs and outputs actually switch places (as well as  the matrices governing the inputs and outputs). Note that I've left   out of this for now. I'm not really sure yet of a good way to  fit it into this picture; it's possible that  is just unnatural  mathematically but sometimes necessary physically (although usually we  can assume that ).
Now that we  have this nice framework for thinking about linear control systems, I'm  going to introduce controllers and observers, and it will be easy to see  that they are dual to each other in the sense just described.
Controllability  and Observability
Go back to  the non-linear case for a moment and suppose that we have a system  , or, in the notation I've been using, . We say that such a system is controllable if for  any two states  and , there exists a time  and a control signal  such that if  then  when the system is driven by the control  signal . What this says intuitively is that we can get from  any state to any other state in a finite amount of time.
For linear  systems, controllability implies something stronger — we can actually  get from any state to any other state arbitrarily quickly, and this is  often times the definition given in the linear case. For non-linear  systems, this is not the case, as a trivial example we could have

There are a  few important properties of linear systems that are equivalent to  controllability:
(1) There is  no proper subspace  of the state space such that  and , where  is the space of  possible instantaneous control signals. The intuition is that there is  no subspace that the passive dynamics (without control) can get stuck in  such that the control input can't move the dynamics out of that space.
(2) There is  no left eigenvector of  that is in the left null space of  . In other words, it actually suffices to check the criterion  (1) above just for one-dimensional subspaces.
(3) The  matrix , where  is  the dimension of the state space of the system, has full row rank.
(4) For any  choice of  eigenvalues ,  there exists a matrix  such that  has generalized  eigenvalues . We can think of this as saying that an appropriate linear feedback law  can be used to give the closed-loop (i.e. after control is applied) dynamics arbitrary eigenvalues.
I will leave  (1) and (2) to you as exercises. Note that this is because I actually  think you can solve them, not because I'm being lazy. (3) I will prove  shortly (it is a very useful computational criterion for testing  controllability). (4) I will prove later in this post. I should also  note that these criteria also hold for a discrete-time system

Proof of (3):  In the case of a discrete-time system, if we have control inputs , then  will be

In  particular, after  time steps we can affect  by  an arbitrary linear combination of elements from the row spaces of  , where  ranges from  to . In  other words, we can drive  to an arbitrary state if and  only if the row space of  is the entire  state space, i.e.  has full row rank. So a  discrete-time system is controllable if and only if  has full row rank for some sufficiently large  .
To finish the  discrete-time case, we use the Cayley-Hamilton  theorem, which shows that any  matrix satisfies a  degree  polynomial, and so in particular it suffices to pick   above, since  can be written as a linear  combination of  for , and similarly for  any larger powers of .
Now we need  to deal with the continuous time case. In this case, we can use the  theory of linear differential equations to show that

where  is the matrix  exponential of . But if we use the Cayley-Hamilton  theorem a second time, we see that $e^{A	au}$ can be expressed as an  st degree polynomial in , so that there exists  some  such that

From here it is clear that, in order for a continuous time system to  be controllable, the controllability matrix must have full row rank  (since  is equal to  plus something in the  row space of the controllability matrix). The converse is less obvious. If the  were linearly independent functions, then we would be done, because the last term in the sum can be thought of as the inner product of  and , and we can just use Gram-Schmidt orthogonalization to show that those inner products can be chosen arbitrarily (if you don't see this then figuring it out is a good linear algebra exercise).
The problem is that the  are not necessarily linearly independent. If  has all distinct eigenvalues, then they will be. This is because we have the relations  and  for any -eigenvector  of , so we can write  distinct exponential functions as a linear combination of the , and any relation among the  would imply a relation among the , which is impossible (it is a basic result from Fourier analysis that exponential functions are linearly independent).
However, this result actually needs  to have distinct eigenvalues. In particular, if one takes , the  identity matrix, then you can show that all but one of the  can be chosen arbitrarily. This is because , ,  are all equal to each other, and thus linearly dependent.
What we need to do instead is let  be the degree of the minimal polynomial  such that . Then we can actually write  as  for some functions :

By the way in which the  were constructed (by applying polynomial relations to an absolutely convergent Taylor series), we know that they are all infinitely differentiable, hence we can differentiate both sides  times and write

Now look at these derivatives from  to . If the  were linearly dependent, their derivatives would satisfy the same relation, and therefore (by evaluating everything at , the matrices  would satisfy a linear relation, which is impossible, since then  would satisfy a polynomial relation of degree less than .
So, the  are linearly independent, and thus by the argument with Gram-Schmidt above we can write anything in the row space of  as

for any . So are we done? Almost. The last step we need to finish is to note that if  satisfies a polynomial of degree  then the row space of  is the same as the row space of , for .
So, that proves the result (3) about the controllability matrix. It was a lot of work in the continuous time case, although it matches our intuition for why it should be true (taking an exponential and taking a derivative are somewhat complementary to each other, so it made sense to do so; and I think there are probably results in analysis that make this connection precise and explain why we should get the controllability result in the continuous case more or less for free).
As I said before, (4) will have to wait until later.
In addition to controllability, we have a notion of stabilizability, which means that we can influence all unstable modes of . In other words, we can make sure that the system eventually converges to the origin (although not necessarily in finite time). Versions of criteria (2) and (4) exist for stabilizable systems. Criterion (2) becomes a requirement that no left eigenvector of  whose eigenvalue has non-negative real part is in the left null space of . Criterion (4) becomes a requirement that there exist  such that  has only eigenvalues with negative real part.
Observers
We say that a system is observable if, for any initial state  and any control tape , it is possible in finite time to infer  given only  and the output . In particular, we are not given any information about the internal states  of the system (except through ), although it is assumed that , , , and  are known. If we have a non-linear system

then it is assumed that  and  are known.
It turns out that observability for a system is exactly the same as controllability for the dual system, so all the criteria from the previous section hold in a suitably dual form. One thing worth thinking about is why these results still hold for any control tape .
(1) There is no non-zero subspace  of  such that  and . In other words, there is no space that doesn't show up in the output and such that the natural dynamics of the system stay in that space.
(2) There is no right eigenvector of  that is in the right null space of .
(3) The matrix  has full column rank.
(4) The eigenvalues of  can be assigned arbitrarily by an appropriate choice of .
Just as the matrix  from the previous section can be thought of as a linear feedback law that gives the system arbitrary eigenvalues, the matrix  is part of a feedback law for something called a Luenburger observer.
Also, just as there is stabilizability for a system, meaning that we can control all of the unstable modes, there is also detectability, which means that we can detect all of the unstable modes.
Luenburger Observers
An observer is a process that estimates the state of an observable system given information about its outputs. If a system is detectable, and  is such that  has only eigenvalues with negative real part, then consider the system

Using the fact that , we see that
, so that  decays exponentially to zero (by the assumption on the eigenvalues of . Thus the dynamical system above, which is called a Luenburger observer, will asymptotically approach the true state of a system given arbitrary initial conditions.
If a system is both controllable and observable, can we design an observer and a controller that working together successfully control the system? (This question is non-trivial because the controller has to use the estimated state from the controller, rather than the actual state of the system, for feedback.) The answer is no in general, but it is yes for linear systems.
Let $F$ be such that  is stable and let  be such that  is stable. (A matrix is stable if all of its eigenvalues have negative real part.) Now we will consider the system obtained by using  as a Luenburger observer and  as a linear feedback law. Let . Then we have

In matrix form, this gives

Because of the block triangular form of the matrix, we can see that its eigenvalues are given by the eigenvalues of  and . Since  and  are both stable, so is the matrix given above, so we can successfully stabilize the above system to the origin. Of course, this is weaker than full controllability. However, if we have full controllability and observability, then we can set the eigenvalues of the above matrix arbitrarily, which should imply full controllability (I haven't sat down and proved this rigorously, though).
So, now we know how to stabilize a linear system if it is detectable and stabilizable. The main thing to take away from this is the fact that the poles of the coupled dynamics of state and observation error are exactly the eigenvalues of  and  considered individually.
State-space representations
The final topic I'd like to talk about in this post is state-space representations of transfer functions. It is here that I will prove all of the results that I promised to take care of later. There are plenty more topics in linear control theory, but I've been writing this post for a few days now and it's at a good stopping point, so I'll leave the rest of the topics for a later post.
A state-space representation of a transfer function is exactly what it sounds like. Given a transfer function  from  to , find a state-space model

that has  as a transfer function. We'll be concerned with linear state-space representations only.
The first thing to note is that a linear state-space representation of  can always be reduced to a smaller representation unless the representation is both controllable and observable (by just restricting to the controllable and observable subspace).
The next thing to note is that, since the transfer function of a state-space representation is , a transfer function  has an irreducible (in the sense of the preceding paragraph) linear state-space representation of degree  if and only if , where  and  are polynomials with . Thus all controllable and observable linear state-space representations of  have the same dimension, and therefore there exists some non-canonical vector space isomorphism such that we can think of any two such representations as living in the same state space (though possibly with different matrices , , , and ).
Finally, if two state-space representations over the same vector space have the same transfer function, then one can be obtained from the other by a chance of coordinates. I will now make this more precise and also prove it.
Claim: Suppose that  and  are two (not necessarily linear) state-space representations with the same input-output mapping. If  is controllable and  is observable, then there is a canonical map from the state space of  to the state space of . If  is observable, then this map is injective. If  is controllable, then this map is surjective. If  and  are both linear representations, then the map is linear.
Proof: Let the two representations be  and .
Since  is controllable, we can take an input tape that sends  to an arbitrary state  at some time . Then by looking at  evolve under the same input tape, by the observability of  we will eventually be able to determine  uniquely. The canonical map sends the  we chose to .The fact that  for all  guarantees that  is well-defined (i.e., it doesn't matter what  we choose to get there).
If  is controllable, then we can choose a  that causes us to end up with whatever  we choose, which implies that the map is surjective. Now for the purposes of actually computing the map, we can always assume that the control input becomes  once we get to the desired . Then there is a one-to-one correspondence between possible output tapes after time  and possible values of . If  is observable, this is also true for , which implies injectivity. I will leave it to you to verify that the map is linear if both representations are linear.
Finally, I would like to introduce a special case of controllable canonical form and use it to prove criterion (4) about controllability. It will also show, at least in a special case, that any transfer function that is a quotient of two polynomials (where the denominator has at least as high degree as the numerator) has a linear state-space representation.
The special case is when  is one-dimensional. Then our transfer matrix can be written in the form

It turns out that this transfer function can be represented by the following transfer matrix:

This might seem a bit contrived, but the construction for  is a nice trick for constructing a matrix with a given characteristic polynomial. Also note that $A$ will have a single Jordan block for each distinct eigenvalue (whose size is the number of times that   eigenvalue appears in the list ). One can show directly that this is a necessary and sufficient condition for being controllable by a single input.
I will leave it to you to check the details that the above state-space model actually has  as a transfer function. (Bonus question: what is the equivalent observable canonical form for observable single-output systems?) I will wrap up this post by proving criterion (4) about controllability, as promised. I have reproduced it below for convenience:
(4) An LTI system is controllable if and only if we can assign the eigenvalues of  arbitrarily by a suitable choice of .
Proof: I will prove the "only if" direction, since that is the difficult direction. First consider the case when we have a single-input system. Then take the transfer function from  to  (this is the same as assuming that , ). By the result above and the assumption of controllability, there exists a system with the same transfer function in controllable canonical form, and thus there is a change of coordinates that puts our system in controllable canonical form. Once we are in canonical form, it is easy to see that by choosing , we end up with a system whose characteristic polynomial is . We can therefore give  an arbitrary characteristic polynomial, and thus choose its eigenvalues arbitrarily.
This proves the desired result in the case when we have a single input to our system. When we have multiple inputs, we have to consider them one-by-one, and use the fact that linear feedback can't affect the eigenvalues of the parts of the system that are outside the controllable subspace. I haven't checked this approach very carefully, so it might not work, but I am pretty sure it can be made to work. If you want more details, feel free to ask me and I will provide them. At this point, though, I'm writing more of a treatise than a blog post, so I really think I should cut myself off here. I hope the exposition hasn't suffered at all from this, but if it has, feel free to call me on it and I will clarify myself.
My next post will take a break from linear control and tell you why using least squares is one of the worst ideas ever (because you think it will work when it actually won't; if you don't believe me I'll show you how negligible sampling errors can easily cause you to be off by 10 percent in your model parameters).

Linear Control Theory: Part I


The Underwater Cartpole

My last few posts have been rather abstract. I thought I'd use this one to go into some details about the actual system we're working with.
As I mentioned before, we are looking at a cart pole in a water tunnel. A cart pole is sometimes also called an inverted pendulum. Here is a diagram from wikipedia:
The parameter we have control over is F, the force on the cart. We would like to use this to control both the position of the cart and the angle of the pendulum. If the cart is standing still, the only two possible fixed points of the system are  (the bottom, or "downright") and  (the "upright"). Since  is easy to get to, we will be primarily interested with getting to .
For now, I'm just going to worry about the regular cart pole system, without introducing any fluid dynamics. This is because the fluid dynamics are complicated, even with a fairly rough model (called the Quasi-steady Model), and I don't know how to derive them anyway. Before continuing, it would be nice to have an explicit parametrization of the system. There are two position states we care about: , the cart position; and , the pendulum angle, which we will set to  at the bottom with the counter-clockwise direction being positive. I realize that this is not what the picture indicates, and I apologize for any confusion. I couldn't find any good pictures that parametrized it the way I wanted, and I'm going to screw up if I use a different parametrization than what I've written down.
At any rate, in addition to the two position states  and , we also care about the velocity states  and , so that we have four states total. For convenience, we'll also name a variable , so that we have a control input  that directly affects the acceleration of the cart. We also have system parameters  (the mass of the cart),  (the acceleration due to gravity),  (the length of the pendulum arm), and  (the inertia of the pendulum arm). With these variables, we have the following equations of motion:

You will note that the form of these equations is different from in my last post. This is because I misspoke last time. The actual form we should use for a general system is

or, if we are assuming a second-order system, then

Here we are assuming that the natural system dynamics can be arbitrarily non-linear in , but the effect of control is still linear for any fixed system state (which, as I noted last time, is a pretty safe assumption). The time when we use the form  is when we are talking about a linear system — usually a linear time-invariant system, but we can also let  and  depend on time and get a linear time-varying system.
I won't go into the derivation of the equations of motion of the above system, as it is a pretty basic mechanics problem and you can find the derivation on Wikipedia if you need it. Instead, I'm going to talk about some of the differences between this system and the underwater system, why this model is still important, and how we can apply the techniques from the last two posts to get a good controller for this system.
Differences from the Underwater System
In the underwater system, instead of having gravity, we have a current (the entire system is on the plane perpendicular to gravity). I believe that the effect of current is much the same as the affect of gravity (although with a different constant), but that could actually be wrong. At any rate, the current plays the role that gravity used to play in terms of defining "up" and "down" for the system (as well as creating a stable fixed point at  and an unstable fixed point at ).
More importantly, there is significant drag on the pendulum, and the drag is non-linear. (There is always some amount of drag on a pendulum due to friction of the joint, but it's usually fairly linear, or at least easily modelled.) The drag becomes the greatest when , which is also the point at which  becomes useless for controlling  (note the  term in the affect of  on ). This means that getting past  is fairly difficult for the underwater system.
Another difference is that high accelerations will cause turbulence in the water, and I'm not sure what affect that will have. The model we're currently using doesn't account for this, and I haven't had a chance to experiment with the general fluid model (using PDEs) yet.
Why We Care
So with all these differences, why am I bothering to give you the equations for the regular (not underwater) system? More importantly, why would I care about them for analyzing the actual system in question?
I have to admit that one of my reasons is purely pedagogical. I wanted to give you a concrete example of a system, but I didn't want to just pull out a long string of equations from nowhere, so I chose a system that is complex enough to be interesting but that still has dynamics that are simple to derive. However, there are also better reasons for caring about this system. The qualitative behaviour of this system can still be good for giving intuition about the behaviour of the underwater system.
For instance, one thing we want to be able to do is swing-up. With limited magnitudes of acceleration and a limited space (in terms of ) to perform maneuvers in, it won't be possible in general to perform a swing-up. However, there are various system parameters that could make it easier or harder to perform the swing-up. For instance, will increasing  (the inertia of the pendulum) make it easier or harder to perform a swing-up? (You should think about this if you don't know the answer, so I've provided it below the fold.)

The answer is that higher inertia makes it easier to perform a swing-up (this is more obvious if you think about the limiting cases of  and ). The reason is that a higher moment of inertia makes it possible to store more energy in the system at the same velocity. Since the drag terms are going to depend on velocity and not energy, having a higher inertia means that we have more of a chance of building up enough energy to overcome the energy loss due to drag and get all the way to the top.
In general, various aspects of the regular system will still be true in a fluid on the proper time scales. I think one thing that will be helpful to do when we start dealing with the fluid mechanics is to figure out exactly which things are true on which time scales.
What we're currently using this system for is the base dynamics of a high-gain observer, which I'll talk about in a post or two.
I apologize for being vague on these last two justifications. The truth is that I don't fully understand them myself. The first one will probably have to wait until I start toying with the full underwater system; the second (high-gain observers) I hope to figure out this weekend after I check out Khalil's book on control from Barker Library.
Hopefully, though, I've at least managed somewhat to convince you that the dynamics of this simpler system can be informative for the more complicated system.
Controlling the Underwater Cartpole
Now we finally get to how to control the underwater cartpole. Our desired control task is to get to the point . That is, we want to get to the unstable fixed point at . In the language of my last post, if we wanted to come up with a good objective function , we could say that  is equal to the closest we ever get to  (assuming we never pass it), and if we do get to  then it is equal to the smallest velocities we ever get as we pass ; also,  is equal to infinity if  ever gets too large (because we run into a wall), or if  gets too large (because we can only apply a finite amount of acceleration).
You will notice that I am being pretty vague about how exactly to define  (my definition above wouldn't really do, as it would favor policies that just barely fail to get to  over policies that go past it too quickly, which we will see is suboptimal). There are two reasons for my vagueness — first, there are really two different parts to the control action — swing-up and balancing. Each of these parts should really have its own cost function, as once you can do both individually it is pretty easy to combine them. Secondly, I'm not really going to care all that much about the cost function for what I say below. I did have occasion to use a more well-defined cost function for the swing-up when I was doing learning-based control, but this didn't make its way (other than by providing motivation) into the final controller.
I should point out that the actual physical device we have is more velocity-limited than acceleration-limited. It can apply pretty impressive accelerations, but it can also potentially damage itself at high velocities (by running into a wall too quickly). We can in theory push it to pretty high velocities as well, but I'm a little bit hesitant to do so unless it becomes clearly necessary, as breaking the device would suck (it takes a few weeks to get it repaired). As it stands, I haven't (purposely) run it at higher velocities than 1.5 meters/sec, which is already reasonably fast if you consider that the range of linear motion is only 23.4 cm.
But now I'm getting sidetracked. Let's get back to swing-up and balancing. As I said, we can really divide the overall control problem into two separate problems of swing-up and balancing. For swing-up, we just want to get enough energy into the system for it to get up to . We don't care if it's going too fast at  to actually balance. This is because it is usually harder to add energy to a system than to remove energy, so if we're in a situation where we have more energy than necessary to get to the top, we can always just perform the same control policy less efficiently to get the right amount of energy.
For balancing, we assume that we are fairly close to the desired destination point, and we just want to get the rest of the way there. As I mentioned last time, balancing is generally the easier of the two problems because of LQR control.
In actuality, these problems cannot be completely separated, due  to the finite amount of space we have to move the cart in. If the swing  up takes us to the very edge of the available space, then the balancing  controller might not have room to actually balance the pendulum.
Swing-up
I will first go  in to detail on the problem of swing-up. The way I think about this is  that the pendulum has some amount of energy, and that energy gets sapped  away due to drag. In the underwater case, the drag is significant  enough that we really just want to add as much energy as possible. How  can we do this? You will recall from classical mechanics that the faster  an object is moving, the faster you can add energy to that object.  Also, the equations of motion show us that an acceleration in   has the greatest effect on  when  is largest, that is, when  or . At the same time, we expect the pendulum to be moving  fastest when , since at that point it has the smallest  potential energy, and therefore (ignoring energy loss due to drag), the  highest kinetic energy. So applying force will always be most useful  when .
Now there is a slight problem with this  argument. The problem is that, as I keep mentioning, the cart only has a  finite distance in which to move. If we accelerate the cart in one  direction, it will keep moving until we again accelerate it in the  opposite direction. So even though we could potentially apply a large  force at , we will have to apply a similarly large  force later, in the opposite direction. I claim, however, that the following policy is still optimal: apply a large force at , sustain that force until it becomes necessary to decelerate (to avoid running into a wall), then apply a large decelerating force. I can't prove rigorously that this is the optimal strategy, but the reasoning is that this adds energy when  is changing the fastest, so by the time we have to decelerate and remove energy  will be significantly smaller, and therefore our deceleration will have less effect on the total energy.
To do the swing-up, then, we just keep repeating this policy whenever we go past  (assuming that we can accelerate in the appropriate direction to add energy to the system). The final optimization is that, once we get past , the relationship between  and  flips sign, and so we would like to apply the same policy of rapid acceleration and deceleration in this regime as well. This time, however, we don't wait until we get to , as at that point we'd be done. Instead, we should perform the energy pumping at , which will cause  to increase above  again, and then go in the opposite direction to pump more energy when  becomes  for the second time.
I hope that wasn't too confusing of an explanation. When I get back to lab on Monday, I'll put up a video of a matlab simulation of this policy, so that it's more clear what I mean. At any rate, that's the idea behind swing-up: use up all of your space in the -direction to pump energy into the system at maximum acceleration, doing so at  and when  and we are past . Now, on to balancing.
Balancing
As I mentioned, if we have a good linear model of our system, we can perform LQR control. So the only real problem here is to get a good linear model. To answer Arvind's question from last time, if we want good performance out of our LQR controller, we should also worry about the cost matrices  and ; for this system, the amount of space we have to balance (23.4cm, down to 18cm after adding in safeties to avoid hitting the wall) is small enough that it's actually necessary to worry about  and  a bit, which I'll get to later.
First, I want to talk about how to get a good linear model. To balance, we really want a good linearization about . Unfortunately, this is an unstable fixed point so it's hard to collect data around it. It's easier to instead get a good linearization about  and then flip the signs of the appropriate variables to get a linear model about . My approach to getting this model was to first figure out what it would look like, then collect data, and finally do a least squares fit on that data.
Since we can't collect data continuously, we need a discrete time linear model. This will look like

In our specific case,  and  will look like this:

I got this form by noting that we definitely know how , , and  evolve with time, and the only question is what happens with . On the other hand, clearly  cannot depend on  or  (since we can set them arbitrarily by choosing a different inertial reference frame). This leaves only three variables to determine.
Once we have this form, we need to collect good data. The important thing to make sure of is that the structure of the data doesn't show up in the model, since we care about the system, not the data. This means that we don't want to input something like a sine or cosine wave, because that will only excite a single frequency of the system, and a linear system that is given something with a fixed frequency will output the same frequency. We should also avoid any sort of oscillation about , or else our model might end up thinking that it's supposed to oscillate about  in general. I am sure there are other potential issues, and I don't really know much about good experimental design, so I can't talk much about this, but the two issues above are ones that I happened to run into personally.
What I ended up doing was taking two different functions of  that had a linearly increasing frequency, then differentiating twice to get acceleration profiles to feed into the system. I used these two data sets to do a least squares fit on , , and , and then I had my model. I transformed by discrete time model into a continuous time model (MATLAB has a function called d2c that can do this), inverted the appropriate variables, and got a model about the upright ().
Now the only problem was how to choose  and . The answer was this: I made  fairly small (), since we had a very strong actuator so large accelerations were fine. Then, I made the penalties on position larger than the penalties on velocity (since position is really what we care about). Finally, I thought about the amount that I would want the cart to slide to compensate for a given disturbance in , and used this to choose a ratio between costs on  and costs on . In the end, this gave me .
I wanted to end with a video of the balancing controller in action, but unfortunately I can't get my Android phone to upload video over the wireless, so that will have to wait.

The Underwater Cartpole


Linear Control Theory: Part 0

The purpose of this post is to introduce you to some of the basics of control theory and to introduce the Linear-Quadratic Regulator, an extremely good hammer for solving stabilization problems.
To start with, what do we mean by a control problem? We mean that we have some system with dynamics described by an equation of the form

where  is the state of the system and  is some matrix (which itself is allowed to depend on ). For example, we could have an object that is constrained to move in a line along a frictionless surface. In this case, the system dynamics would be

Here  represents the position of the object, and  represents the velocity (which is a relevant component of the state, since we need it to fully determine the future behaviour of the system). If there was drag, then we could instead have the following equation of motion:

where  is the coefficient of drag.
If you think a bit about the form of these equations, you will realize that it is both redundant and not fully general. The form is redundant because  can be an arbitrary function of , yet it also acts on  as an argument, so the equation , for example, could be written as

for any choice of . On the other hand, this form is also not fully general, since  will always be a fixed point of the system. (We could in principle fix this by making  affine, rather than linear, in , but for now we'll use the form given here.)
So, if this representation doesn't uniquely describe most systems, and can't describe other systems, why do we use it? The answer is that, for most systems arising in classical mechanics, the equations naturally take on this form (I think there is a deeper reason for this coming from Lagrangian mechanics, but I don't yet understand it).
Another thing you might notice is that in both of the examples above,  was of the form . This is another common phenomenon (although  and  may be vectors instead of scalars in general), owing to the fact that Newtonian mechanics produces second-order systems, and so we care about both the position and velocity of the system.
So, now we have a mathematical formulation, as well as some notation, for what we mean by the equations of motion of a system. We still haven't gotten to what we mean by control. What we mean is that we assume that, in addition to the system state , we have a control input  (usually we can choose  independently from ),  such that the actual equations of motion satisfy

where again,  and  can both depend on . What this really means physically is that, for any configuration of the system, we can choose a control input , and  will affect the instantaneous change in state in a linear manner. We normally call each of the entries of  a torque.
The assumption of linearity might seem strong, but it is again true for most systems, in the sense that a linear increase in a given torque will induce a linear response in the kinematics of the system. But note that this is only true once we talk about mechanical torques. If we think of a control input as an electrical signal, then the system will usually respond non-linearly with respect to the signal. This is simply because the actuator itself provides a force that is non-linear with its electrical input.
We can deal with this either by saying that we only care about a local model, and the actuator response is locally linear to its input; or, we can say that the problem of controlling the actuator itself is a disjoint problem that we will let someone worry about. In either case, I will shamelessly use the assumption that the system response is linear in the control input.
So, now we have a general form for equations of motion with a control input. The general goal of a control problem is to pick a function  such that if we let  then the trajectory $X(t)$ induced by the equation  minimizes some objective function . Sometimes our goals are more modest and we really just want to get to some final state, in which case we can make  just be a function of the final state that assigns a score based on how close we end up to the target state. We might also have hard constraints on  (because our actuators can only produce a finite amount of torque), in which case we can make  assign an infinite penalty to any  that violates these constraints.
As an examples, let's return to our first example of an object moving in a straight line. This time we will say that , with the constraint that . We want to get to  as quickly as possible, meaning we want to get to  and then stay there. We could have  just be the amount of time it takes to  get to the desired endpoint, with a cost of infinity on any  that violates the torque limits. However, this is a bad idea, for two reasons.
The first reason is that, numerically, you will never really end up at exactly , just very close to it. So if we try to use this function on a computer, unless we are particularly clever we will assign a cost of  to every single control policy.
However, we could instead have  be the amount of time it takes to get close to the desired endpoint. I personally still think this is a bad idea, and this brings me to my second reason. Once you come up with an objective function, you need to somehow come up with a controller (that is, a choice of ) that minimizes that objective function, or at the very least performs reasonably well as measured by the objective function. You could do this by being clever and constructing such a controller by hand, but in many cases you would much rather have a computer find the optimal controller. If you are going to have a computer search for a good controller, you want to make the search problem as easy as possible, or at least reasonable. This means that, if we think of  as a function on the space of control policies, we would like to make the problem of optimizing  tractable. I don't know how to make this precise, but there are a few properties we would like  to satisfy — there aren't too many local minima, and the minima aren't approached too steeply (meaning that there is a reasonable large neighbourhood of small values around each local minimum). If we choose an objective function that assigns a value of  to almost everything, then we will end up spending most of our time wading through a sea of infinities without any direction (because all directions will just yield more values of ). So a very strict objective function will be very hard to optimize. Ideally, we would like a different choice of  that has its minimum at the same location but that decreases gradually to that minimum, so that we can solve the problem using gradient descent or some similar method.
In practice, we might have to settle for an objective function that only is trying to minimize the same thing qualitatively, rather than in any precise manner. For example, instead of the choice of  discussed above for the object moving in a straight line, we could choose

where  is some arbitrary final time. In this form, we are trying to minimize the time-integral of some function of the deviation of  from . With a little bit of work, we can deduce that, for large enough , the optimal controller is a bang-bang controller that accelerates towards $0$ at the greatest rate possible, until accelerating any more would cause the object to overshoot , at which point the controller should decelerate at the greatest rate possible (there are some additional cases for when the object will overshoot the origin no matter what, but this is the basic idea).
This brings us to my original intention in making this post, which is LQR (linear-quadratic regulator) control. In this case, we assume that  and  are both constant and that our cost function is of the form

where the  means transpose and  and  are both positive definite matrices. In other words, we assume that our goal is to get to , and we penalize both our distance from  and the amount of torque we apply at each point in time. If we have a cost function of this form, then we can actually solve analytically for the optimal control policy . The solution involves solving the Hamilton-Bellman-Jacobi equations, and I won't go into the details, but when the smoke clears we end up with a linear feedback policy , where , and  is given by the solution to the algebraic Riccati equation

What's even better is that MATLAB has a built-in function called lqr that will set up and solve the Riccati equation automatically.
You might have noticed that we had to make the assumption that both  and  were constant, which is a fairly strong assumption, as it implies that we have a LTI (linear time-invariant) system. So what is LQR control actually good for? The answer is stabilization. If we want to design a controller that will stabilize a system about a point, we can shift coordinates so that the point is at the origin, then take a linear approximation about the origin. As long as we have a moderately accurate linear model for the system about that point, the LQR controller will successfully stabilize the system to that point within some basin of attraction. More technically, the LQR controller will make the system locally asymptotically stable, and the cost function  for the linear system will be a valid local Lyapunov function.
Really, the best reason to make use of LQR controllers is that they are a solution to stabilization problems that work out of the box. Many controllers that work in theory will actually require a ton of tuning in practice; this isn't the case for an LQR controller. As long as you can identify a linear system about the desired stabilization point, even if your identification isn't perfect, you will end up with a pretty good controller.
I was thinking of also going into techniques for linear system identification, but I think I'll save that for a future post. The short answer is that you find a least-squares fit of the data you collect. I'll also go over how this all applies to the underwater cart-pole in a future post.

Linear Control Theory: Part 0


Robotics

This summer I am working in the Robotics Locomotion group at CSAIL (MIT's Computer Science and Artificial Intelligence Laboratory). I've decided to start a blog to exposit on the ideas involved. This ranges from big theoretical ideas (like general system identification techniques) to problem-specific ideas (specific learning strategies for the system we're interested in) to useful information on using computational tools (how to make MATLAB's ode45 do what you want it to).
To start with, I'm going to describe the problem that I'm working on, together with John (a grad student in mechanical engineering).
Last spring, I took 6.832 (Underactuated Robotics) at MIT. In that class, we learned multiple incredibly powerful techniques for nonlinear control. After taking it, I was more or less convinced that we could solve, at least off-line, pretty much any control problem once it was posed properly. After coming to the Locomotion group, I realized that this wasn't quite right. What is actually true is that we can solve any control problem where we have a good model and a reasonable objective function (we can also run into problems in high dimensions, but even there you can make progress if the objective function is nice enough).
So, we can (almost) solve any problem once we have a good model. That means the clear next thing to do is to come up with really good modelling techniques. Again, this is sort of true. There are actually three steps to constructing a good controller: experimental design, system identification, and a control policy.
System identification is the process of building a good model for your system given physical data from it. But to build a good model, you need good data. That's where experimental design comes in. Many quick and dirty ways of collecting data (like measuring the response of a sinusoidal input) introduce flaws into the model (which cannot be fixed except by collecting more data). I will explain these issues in more detail in a later post. For simple systems, you can get away with still-quick but slightly-less-dirty methods (such as a chirp input), but for more general systems you need better techniques. Ian (a research scientist in our lab) has a nice paper on this topic that involves semidefinite optimization.
Designing a control policy we have already discussed. It is the process of, once we have collected data and built a model, designing an algorithm that will guide our system to its desired end state.
So, we have three tasks — experimental design, system identification, and control policy. If we can do the first two well, then we already know how to do the third. So one solution is to do a really good job on the experimental design and system identification so that we can use our sophisticated control techniques. This is what Michael and Zack are working on with a walking robot. Zack has spent months building the robot in such a way that it will obey all of our idealized models and behave nicely enough that we can run LQR-trees on it.
Another solution is to give up on having a perfect model and have a system identification algorithm that, instead of returning a single model, returns an entire space of models (for example, by giving an uncertainty on each parameter). Then, as long as we can build a controller that works for every system in this space, we'll be good to go. This can be done by using techniques from robust control.
A final idea is to give up on models entirely and try to build a controller that relies only on the qualitative behaviour of the system in question (for example, by using model-free learning techniques). This is what I am working on. More specifically, I'm working on control in fluids with reasonably high Reynold's number. Unless you can solve the Navier-Stokes equations, you can't hope to get a model for this system, so you'll have to do something more clever.
The first system we're working with is the underwater cart-pole system. This involves a pendulum attached to a cart. The pendulum itself is not actuated (meaning there is no motor to swing it up). Instead, the only way to control the pendulum is indirectly, by moving the cart around (the cart is constrained to move in a line). The fluid dynamics enter when we put the pendulum in a water tunnel and replace the arm of the pendulum with a water foil.
When the pendulum is in a constant-velocity stream, the system becomes a cart and pendulum with non-linear damping. However, when we add objects to the stream, the objects shed vortices and the dynamics become too complicated to model with an ordinary differential equation. Instead, we need to simulate the solution to a partial differential equation, which is significantly more difficult computationally.
Our first goal is to design a controller that will stabilize the pendulum at the top in the case of a constant stream (we have already done this — more about that in a later post). Our next goal is to design a controller to swing the pendulum up to the top, again in a constant stream (this is what I hope to finish tomorrow — again, more details in a later post). Once we have these finished, the more interesting work begins — to accomplish the same tasks in the presence of vortices. If we were to use existing ideas, we would design a robust version of the controller for a constant stream, and treat the vortices as disturbances. And this will probably be the first thing we do, so that we have a standard of comparison. But there are many examples in nature of animals using vortices to their advantage. So our ultimate goal is to do the same in this simple system. Since vortices represent lots of extra energy in the water, our hope is to actually pull energy out of the vortex to aid in the swing-up task, thus actually using less energy than would be needed without vortices (if this sounds crazy, consider that dead trout can swim upstream using vortices).
Hopefully this gives you a good overview of my project. This is my first attempt at maintaining a research blog, so if you have any comments to help improve the exposition, please give them to me. Also, if you'd like me to elaborate further on anything, let me know. I'll hopefully have a post or two going into more specific details this weekend.

Robotics