On 30/08/2013 06:12, Default User wrote: > Posted to both sci.math.num-analysis and sci.math.symbolic based on a > suggestion in sci.math. I will follow both groups if you'd like to > remove the crosspost. > > > I'm working on bit of code that does some data processing. I'd like > some input on developing the algorithm. This will effectively condense > a large set of data values to a smaller one. > > The original data is the result of sampling a waveform. The processed > data should still represent that waveform, of course.
Are you sure that the waveform harmonic content will cooperate with that much downsampling?
Do you really want a basic boxcar average here or a low pass filter? > > Input: 10,000 integer values. > Output: 1023 integer values. > > There's nothing that can be done to change the input or output > requirements.
Why reduce the raw data to exactly 1023 output values? Is it to make Fourier analysis particularly difficult?
1023 = 3x11x31 is a Mersenne number but why do you need it? > > My current method is a bit brute force, using a moving window that > sometimes averages 10 raw values, sometimes 9, to derive the results. > > I'm not overly happy with that, so if anyone has some more elegant or > efficient suggestions I'd be quite interested.
You could do a convolutional code that subsamples (and/or low pass filters) from anything down to anything but whether or not it is meaningful to do so depends critically on your data and what you intend to do with it afterwards. Look for the gridding functions for radio astronomy as a 1D variant of that code will probably do what you want. It will be quite clumsy for such non commensurate data lengths.
> I had looked at data compression algorithms. I think those are really > overkill for what I need, plus some problems. My situation is not a > round-trip one. That is, it's not the case that I need to compress, > store or transmit, and then uncompress later. > > The 1023 data points are the final format for the receiving device. So > I have to end up with that number of data points representing the > entire sampled waveform. The instrument can only provide certain > numbers of points.
Can't you throw away a few points at the end of the buffer? Or (zero) pad the reduced data to make it 1023 in length?
> It would be convenient if I could get the raw data as 1023 points, but > that's not an option. In the range we want to work in, I can get 1000 > or 10,000. So we go up a range and average the results. > > > > Brian >