Yet another controller board

Ther is currently no real reason why I started this project other than "because I can". It basically receives data from FlyPT Mover and converts that to step/direction signals to the motor drivers. So nothing new that some other controller couldn't also do, for example the sim4motion board. But I'd like to have a bit more control over some features and I see it as a challange to try to get better performance. And I don't feel comfortable doing a "beggars dance" to get answers to my admittedly numerous and possibly annoying questions. So I decided to build my own.

It might look a bit ugly and over-complicated. To avoid having to build everything from scratch I took the motherboard from a CNC controller I've already designed and only changed the daughter boards. Originally there were stepper motor drivers. I replaced them by small boards with differential line drivers which are connected to the servo drives with RJ45 patch cables for the step/dir signals.
Some specifications:

• 24V power supply
• 8-core 200MHz CPU
• 100Mb/s Ethernet port for UDP connection
• up to 5 axes, 2MHz max. step frequency (new: now 6 axers)
  
• 4 outputs, 9 inputs 24V
• safety support (Estop, STO and brakes)
• OLED status display

I know, 5 axes are too few for most users. But that's a limitation of the current motherboard, not the processor. The CPU has plenty of resources left over (pins, timers...) so as soon as I need it I can make a bigger board.
Today, I ran the first test. I can receive data from FlyPT mover and move one motor with the sliders in the RIG window. Homing, moving to neutral position and parking also works.

Sweet project! I'm curious. If you had to use 10 axis, would you get better performance sending 10 channels out from mover to a custom version of your board or would it be faster for mover to send 5 channels out to two of your boards?

By better performance I suppose latency would be the highest priority.

Cool, What software are you using? Is it targeted for CNC? LinuxCNC? Mach3, your own? That is a good looking controller.

I am heading down this motor controller path and I am thinking about using Arduino boards and the code created by zhai1987. I have found some motor controller designs out there, some that claim to be open source, like Opensfx100, but are not really. I'm still looking.....

Cubexxx has open and free code for step dir control of servos. He even wrote a version for continuous rotation if that does anything for you.

Great Project please report your progress. i Enjoyed to answer your "numerous and possibly annoying questions"

It doesn't really matter. Sending a small packet over Ethernet takes 5 or maybe 10µs. But if I eventually decide to make a commercial project out of it (which I don't at the moment) then the question "how many axes per board?" is important for economical reasons. You don't want to force half of the users to buy two boards. You also don't want to make the board twice as expensive because of spare connectors most of the users never need. So I'd think 7 axes is a good compromise. Only few people need more and if the do then buying a second board is acceptable.

As I said, the CPU has lots of resources. It has 64 IO pins without any restrictions. Any of those 64 pins can be programmed as PWM output, ADC input or whatever you want.

Most of the latency is caused by the operating system. As somebody else said here in the forum "you can ask windows to do 'something' every 2ms but from time to time windows decides to do 'something else'." As soon as the data packet leaves the PC the further delays down the path are no problem. The motion controller and the servos have no OS but run in real time. And any latency <20ms is not noticed by human senses and therefore doesn't matter anyway.

BUT, and this is a big one, jitter is a problem. 20ms delay is no problem as long it is constant. But 1 or 2ms variations of the delay can ruin your calculations as I described here. For CNC applications this is even more important. Old controllers like Mach3 over LPT port in combination with stepper motors had this problem.

Mach3 (and in general all step/dir solutions using software pulse generation) had a basic interrupt period of 25 or 45kHz. In every interrupt it can only decide to output a pulse or not. So if you have to output 22.5kHz it outputs a pulse every second interrupt. Fine. But how do you generate 30kHz? It has to output 1-1-0 pulses and repeat. That makes 30kHz on average but in reality it's 45kHz with a gap in between. This causes the motor to accelerate and decellerate all the time. It sounds noisy and can even stall the motor if a resonance frequency is hit.

Servos are much more forgiving. They do not stall or loose steps even if there are small discontinuities in the frequency of the step signal. In fact, cheap chinese servos "out of the shelf" are very forgiving because the default PID gain settings for the control loop are low. This smoothens out the jitter in the signals which is good but it also makes the response sloppy.

Long explanation short, low jitter and high quality signals are my main priority. And safety, and reliabilty of course. I'm maybe a bit spoiled because of my industrial background.

Thanks. I can't talk much about the CNC project, at least not with this board. It's work in progress for a customer. But I already sell other CNC controllers with my own software. If you're interested check out this.

About the simulator version... I consider making it open source once it's finished. But you know, projects like this are never really finished. I mean, if it turns out to be an advantage over other solutions and there is some demand.

I use the Propeller 2 from parallax. It's a really "special" controller which was designed by a hardware hacker especially for the purpose of hardware hacks. It doesn't have any standard peripherals built in but has very powerful assembler instructions. It doesn't have a dedicated Ethernet controler but instead bit-bangs the RMII protocol at 100Mb/s! The downside is that there is very little "library" code for this processor. I mainly write everything by myself and from scratch. But once the low level drivers are there you can program it in C just like any other processor.

Currently, I only support FlyPT Mover. But I think any motion cueing software that can send UDP data could be used. And I don't have plans to calculate reverse kinematics inside my controller. Not because that would be a problem but simply because it's not needed. I think it's better to do that on the PC side.

@Aerosmith Yes, we talked about your servo system, and thanks for the interesting insights from an expert like you.

As you were reporting here:
https://www.xsimulator.net/community/threads/flypt-mover.13464/page-188#post-249600 .

I would like to join the discussion on this topic as I am also interested in it.

Most people using the AASD servo do not have problems with "jitter" in the position pulse instruction mode. The servos run quite smoothly with an electronic gear ratio of 1:2. It feels like you are really gliding through the air in flight simulators. The jitter comes from the PC, as you can't send messages faster than every 2 ms with a certain accuracy. It is possible to block a thread and cause high CPU load. This is a problem with Windows due to the shortest possible thread switching time of about 1 ms. Most sims and games send there data every 15 to 20ms so we use the time in between to caluculate the accelerations and filter them and produce "new" frames every 2ms
The M4s controller sends the delta pulse packages every 2 ms when new data arrives. Sure, you can now wait for the next packages to arrive and perform smoothing and calculate a speed, but that only works as a filter again.

So, what idea do you have to solve the jitter issue caused around the 2ms windows not exact timing problem without increasing the filter time? i did test with the m4s controller to send every 1ms and also calculate extra filtered frames to send more regular. but with aasd it doesnt matter.

The AASD servo uses the internal filter with 30 ms for pulse instructions and 10 ms for the feedforward filter with 20% gain, and this works well. I tested with electronics gearing 1:1 (only with excellent EMI protection) and was able to switch off the internal AASD filters without having problems with the smoothness of the rig.

Hello Denis, (@Motion4Sim ),
don't get me wrong. This is not intended as complaint or critizism. I have no doubt that most users have no problems with the AASD servos.

Ah, this explains a lot. As I said I already know that the default settings often have low gain and high filter settings so nothing goes wrong when the motor is connected the first time. If a user chooses to increase the gain and gets violent vibrations then it's his fault and nobody can blame the manufacturer.

But I come from a different world. For CNC applications you need the servos to be tuned as stiff as possible. There is also a community of DIY builders for CNC machines. Most people who build a machine for the first time know nothing about servo tuning. Then they wonder why they can't machine really round circles but get "eggs", ellipses or even "potatoes". Then they reduce the feedrate and the cutting depth or cutter diameter to ridicolously low values to improve precision. They stare in disbelief when I show them that the same machine can run at 4m/min with a 8mm cutter full depth through a 10mm aluminium plate and produce a nice shiny surface with 0.02mm tolerance if only the servos are tuned perfectly.

But of course, for a simulation rig this is not only completely unnecessary but in fact counterproductive. Nobody cares if the position is 1mm off. Even 10mm will most likely not be noticed at all. All that counts is that the vector of the acceleration force (including gravity) points in the right direction and has more or less an absolute value that fits the situation of the simulated race or flight.

Maybe I'm a "control freak" but not in the original meaning. This term is most commonly used for machos who enjoy having power over other people and let them do what they tell them. I'm the exact opposite. I don't feel good when I have to ask somebody to implement my ideas or to change their products so they better fit my taste. Instead, I rather do it on my own. This way I have full control over every detail and I also understand everything much better than by reading manuals or trial and error. Of course, it's often more painful and costly than to just "shut up and buy something".

This is a good reason that justifies builting my own controller. This way I can look inside and see what's going on. I can run the debugger or record log files. In my youth I also programmed some games none of which were successful but it's good for learning how realtime systems work. Most games to all timing based on something like the vertical blank interrupt of the screen which has a 50 or 60Hz period. The human brain can't process information faster so a higher update rate makes no sense, at least not for the control inputs and visual output on the screen. Some TVs interpolate between those 50Hz frames to get 100 or even 200 FPS but that's only to reduce stroboscope effects like a fast football producing a series of discrete light spots instead of a continous sweep.

A higher update rate for the servo speed or step/dir signals makes perfect sense, though. If you update the speed signal only once per 20ms you'd get acceleration discontinuities which cause jolts or clicking noises at the transition from one time frame to the next. Explanation here

Yes but there are different ways to interpret the data. The key is to make use of all information that is available. If you have timing information plus position information it's more useful than just position information alone. You can calculate the exact speed if you know the exakt time when the position was sampled even when there is some data transmission delay.

I've made a short video. Sorry, the audio is very soft. If you don't understand my numbling, feel free to ask.

I've been thinking about this for some time... So the best thing we can do is this:

We get position data from the sim software, say every 16ms (black points marked 1,2,3). As the real world is analogue and continous we have to interpolate between thise discrete points somehow. The problem is we can't see into the future. So at the moment we get point #2 we can calculate the current speed and extrapolate that into the interval from 2 to 3. But we do not know the value of #3 until that time has passed and we get the actual value.
So at the time of #3 we find out that our "guess" (the blue line) was wrong and we lag behind and have to catch up. It doesn't matter if we have some sort of "intelligent" algorithm or just apply a numerical filter. The result is a curve that has some unavoidable time delay and ideally has no sharp corners but instead smooth bends. All we can do is to try not to make it worse than necessary.

As I already pointed out in the FlyPT Mover thread if we are not careful jitter can have a bad effect. For example, if the 16ms interval is divided into 8 times 2ms at first it looks like an improvement (green line). But if those additional data points are transmitted with unpredictable delay of +/- 1m what we actually get is the red line. The position deviations don't seem very big but look at the slope angles! The angle = rate of change of position = velocity can change by a factor of 3:1 or more!

The irony in this is that the better the servos are the worse the control loop behaviour becomes. Cheap servos only count pulses and apply heavy filtering to reduce aliasing effects caused by that quantisation. There reaction time is so long they don't even notice the jitter and noise in the velocity. But good servos have a high bandwidth (>1kHz) and calculate velocity feed forward by measuring the frequency of the command input as inverse of the time between pulses. This results in a fast reaction time, low following error and stiff response. But it also makes them sensitive to signals of bad quality.

Anyway... Transmitting data packets at a rate close to the timing jitter and transmission delay (2ms vs. 1ms) is no good idea, IMHO. We have to smoothen the errors out later, anyway. So the high data rate is only a waste of bandwidth and CPU power. I think this is why your turkish customer with the big Boeing cockpit got better results after reducing the data rate.

Using "high quality" servo systems for sim rigs seems also like a waste of money. They might have some advantages like probably making additional transducers (bass shakers) unnecessary as they have enough bandwidth well into the audible range and could take over that task. But it's most likely not worth the trouble.

Thank you for the demonstration. Such behavior was what I expected, that the data is always ahead of the motor positions and the servo or Stepper driver, since there is also an MCU inside, calculates that.

Our goal is therefore to send continuous data sets without jitter, and our approach is to send a pulse train to the current position.

Yes, and that's exactly what we do. The M4S sends the upcoming pulses every two milliseconds to the drive and thats it. So we can also create bass shakers with the AASD drivers with frequencies up to 500 Hz without any issue. i know there are other drivers in the market can do lower but they also cost a lot more money.

What I have to admit now is that the M4S code always waits for the packet from FlyPT before incorporating it into the calculation. Therefore, there may be a kind of jitter in that process, it is.
possible.

But the issue is also known to me, and I have routines running in M4S that do not wait for the signal from FlyPT and continuously incorporate the incoming data with a filter algorithm. but 2 years ago as i was testing this on my sim there was no difference in the feeling so i didnt change it.

But as I mentioned before, my approach was to split the signal more finely and smooth it out, which is why we send the pulses every 2 milliseconds instead of every 16 milliseconds. And it works as best with the servos used from the community here.

the m4s generates the pulse train with the hardware timers/pwm inside the mcu (enable hardwaretimeimg for higher frequencies) otherwise its interrupt driven. Pulses generated by hardware timers are as precise as it gets.

If we send the pulses with a higher delay maybe the simflying comunity will have smoother sailing feeling in long moves, the sim racers will not be satisfied, as they will lose details in the motion they like it really hard .

I have to explain that in a bit more detail. There is currently no feedback from the actual position of the motor. That is processed in the servo drive, only (and not at all for stepper motors).

What you see in the scope display are the green line representing the commanded position from FlyPT Mover and the red line representing the output position to the drive. Servos may have additional following error which is not shown. Steppers under load also have following error but with the free spinning motor this can be neglected (<1.8° and <2ms).

I perform some sort of "safety monitoring", that is, I limit the speed and acceleration to predefined maximum values. So if those values are set reasonably the motors can never stall or overload. That is similar to the spike filtering of FlyPT Mover or M4S controller. As long as the velocity and acceleration is within the limits the output follows the input without any delay or position difference. If the limits are exceeded the output is slowed down and begins to lag behind. You see this as the red and green lines no longer match. The red line tries to catch up as soon as possible with maximum acceleration.

You can notice this the most clearly during the parking motion. The nominal position (green) jumps immerdiately to the parking position (theoretical acceleration = infinite). Of course, the actual position (red) can't do that but followes slowly and smoothly in a nice S-shaped ramp curve.

I currently do it like this: There are two seperate threads (multitasking processes).

1. One processes the input and one the output. The output thread runs once per millisecond and uses the current position and the last calculated velocity to interpolate the output position. If no update to the velocity has been made (which is true at least every second time) the last value is used. An additional moving average filter with a 5ms window smoothens the corners so velocity doesn't change abruptly with infinite acceleration.
   
2. The input process has no fixed period. As soon as a new data packet arrives, current position and velocity are updated. Velocity is calculated as moved distance divided by elapsed time based on the timestamp of the packet.
3. A timeout occurs if no data arrives for 20ms or more. In this case the last position freezes and the nominal velocity is set to zero. This does not cause a excessive jolt as velocity and acceleration are still limited (spike filter).

@Motion4Sim, how do you handle delayed packets? Say, FlyPt Mover commands a constant velocity of 1kHz step pulses. That means at 0ms position is 0 steps, after 2ms it's 2000 steps, after 4ms it's 4000. But let's assume that one packet arrives late, for example position 4000 arrives at 5ms instead of 4ms.

Do you output 2000 steps after the 3ms passed since the last update (2ms)? This would result in a frequency of 2000/3ms = 667Hz. Or do you keep outputting the pulses with a constant time frame every 2ms?

how do you handle delayed packets? Say, FlyPt Mover commands a constant velocity of 1kHz step pulses. That means at 0ms position is 0 steps, after 2ms it's 2000 steps, after 4ms it's 4000. But let's assume that one packet arrives late, for example position 4000 arrives at 5ms instead of 4ms.

It always depends on which protocol is used for data transmission. In our case, we use the serial protocol over the serial interface, and there, all data records are sent one after the other.

What can happen, and I have checked this, it is very very rare, but it happens that occasionally a data record is compromised, meaning that a byte is incorrect. This could lead to errors actually occurring, as checksums are not used so far, except in my own motion software.

In CNC machines, of course, that wouldn't work at all. Every position has to be exactly right.

In our motion, however, it's not as critical if a packet is missing, it will just be filtered out by the lowpass filter we apply, and the data record will disappear. It's actually insignificant.

But to come back to your question, the answer is that it simply doesn't happen that an earlier data record needs to be evaluated later. If that were to happen, when I switch to network protocols soon, it will simply be discarded.

Do you output 2000 steps after the 3ms passed since the last update (2ms)? This would result in a frequency of 2000/3ms = 667Hz. Or do you keep outputting the pulses with a constant time frame every 2ms?[/QUOTE]

Since the positions are filtered anyway, the game with 2, 3, 4, 5 milliseconds is irrelevant. If we use an exponential moving average filter with a factor of 10, for example, the original value will be reached at the earliest after 10 iterations, i.e., after 20 milliseconds.

Then it's also not a problem if one of the values is missing now, as it will only be included in the pulse train chain to the factor of the low-pass filter. This means that if a deviating position with 1000 steps is missing now, at that moment only one-tenth of it would be included in the low

The interpolation is really a good way to obtain the data.

But I do understand what you ultimately mean.
With the motion, we actually want to reach the required position as quickly as possible.
And what is definitely very important is that it has to feel good for the user.

So far, I send out the complete pulse package, the filtered pulse package, every two milliseconds, and the pulse rotation doesn't actually last for the full two milliseconds, as it's mostly finished after the rising edge, because we send out the pulses at >250 kHz and not divided over the time frame.

I had also programmed to divide the pulses to be sent over the transmission period in terms of frequency. But it didn't feel good with the AASD servos, so I discarded that idea.

And I believe that's also the reason why your servos don't run as smoothly with it. The AASD servos, on the other hand, are not bothered by the small gap.

I will now try to operate the AASD servos without a filter and see what I can further optimize in the routine. Perhaps I will also change the pulsing.

Oooops! I thought that the USB protocol makes its own checksums so corrupted data should normally be prevented from ever happening at all. All that you should ever notice is that packets are delayed by 1 or 2 milliseconds because if a false checksum is detected the packet is repeated and that takes some time.

Corrupted data can really screw things up very heavily! A single flipped bit can change the position up to half the total range of the actuator. Even if filtered 1:10 it's still an error of 5% the stroke which will most likely cause the servo to quit due to overcurrent or following error.

So you don't care about the time the packets arrive at all and just output positions as they arrive but with additional filtering. That's totally OK as long as the data transmission is reliable. In the best case scenario I explained above any "more clever" algorithms will not show any real advantages over simple filtering.

So if I understand correctly, you don't spread the pulses over the whole time frame evenly distributed but output them faster so that a gap with no pulses at all is inserted until the next packet arrives and a new pulse burst is started. Ouch! That toitally explains why my servos react so uncomfortable. Of course, if the AASD servos only count pulses and average them out over a time much longer than the burst rate it makes no difference. Ten times short bursts of 250kHz or a constant frequency with evenly spaced pulses result in the same step count after, say 10ms.

But if the servos try to calculate a velocity feed forward from the pulse frequency by measuring the pulse period this must go wrong very miserably.

OK, not your fault, just bad luck. I'll set the velocity feed forward to zero and reduce the position loop gain. And for my next project I use the cheaper servos. I've already bought some.

I just made a drawing of a casing for my board:

We had problems with the 3D printer finishing the casing in time so we had to hook up an improvisation.

The black box on the right side is the M4S controller that is now replaced by my new board in the cardboard box. After some minor problems with one of the servos faulting due to a popped connector we got the homing procedure going and could connect to FlyPT Mover.
WOW! The difference is like day and night.When we connected the controller to the simulator software for the first time I pushed the throttle with the brakes still engaged to make the plane slightly tilt forward. First, I thought it didn't work. I had to look at the motor shaft couplings to see that the servos were actually spinning. Everything was totally silent. We have used the exact same tuning of the servos and the exact same settings in the Mover except that we replaced the serial output with UDP.
But the behaviour is totally different. No more whisteling and grunting noises, no vibrations. Every tuft of grass on the runway could be felt while rolling on the ground and it was a perfectly floating glide while in the air. We had big grins in our faces.

However, there still seems to be a bug in the homing procedure. The parking position above the home switches is multiple cm too high and not equal on all actuators. I have to fix that.

Ok, I've fixed the bug in the homing. I've also changed the state machine so that all axes move simultanously in the same direction but wait until all other axes are in the same state before reversing direction. Before, the homing of each axis was a completely seperate process and synchronisation was only done once after all axes where homed. But this caused unnecessary roll and pitch movements if the positions were uneaqual before the homing.