Accelerometer data game controller

Hi,

I'm wondering if anyone knows how to use data from an accelerometer and map it to the position of an object in hype. Simple experiment for a pong or breakout type game.

https://youtu.be/DMa3RpcPsdY

In the video I use data from the angle which updates the qx innerText to drive style.left and also use the same value to drive rotation, just to tilt a little bit.

var paddle = document.querySelector('#paddle');
var posX = paddle.style.left;

setInterval(function(){
//var newX = document.querySelector('#x').textContent;
var newX = document.querySelector('#qx').textContent;
    paddle.style.left = posX - newX*500 + 'px';		
	hypeDocument.setElementProperty(paddle, 'rotateZ', newX*20)
},20);

The aim however is to use the linear acceleration value of x, but I don't really understand how the accelerator data works to implement it. The movement 'springs' back to the original position. Does anyone have any experience in this area?

Thank you

Matter offers speed and velocity as body-properties.
if you measure one of those over a time you've got a acceleration

https://brm.io/matter-js/docs/classes/Body.html#property_speed
https://brm.io/matter-js/docs/classes/Body.html#property_velocity

to get a physics-body in hype:
hypeDocument.getElementProperty(yourElement, 'physics-body')

It'd be useful for a bit more context or the full code you are using. Judging by your youtube channel, it seems the case is you're not trying to use a mobile phone accelerometer, but an external bluetooth device?

It looks to me though that you're probably getting the delta values, and these should be accumulated over time. (Basically always just add the new device value).

Thanks both @jonathan and @h_classen . I'm happy to post the code, but because it's only usable with the device I'm using, (and because I'm a crappy coder so I'm hacking it together) it probably won't make very much sense.

It is a bluetooth device called MetaWear and I think the values I get from the linear acceleration is in Gs, but I may be wrong. The curve that I can monitor conforms to the explanation this video gives: IMU Data Analysis: Acceleration | Live Video Sync - YouTube

So if the device is moved in positiveX (right?) the curve dips down below zero, then comes up above zero and then settles back down at zero(ish) once device is still. When the device is moved negativeX (left?) then the curve goes up above zero, then goes below zero and then settles at zero when the device is static.

	navigator.bluetooth.requestDevice(
    {filters: [{name: 'MetaWear'}],
      optionalServices: ['0000180a-0000-1000-8000-00805f9b34fb', '0000180f-0000-1000-8000-00805f9b34fb', '326a9000-85cb-9195-d9dd-464cfbbae75a'],
    })  
  .then(device => {
    return device.gatt.connect();
  })
  .then(server => {
    return server.getPrimaryServices();
  })
  .then(services => {
    services.forEach(service => {
        service.getCharacteristics().then(characteristics => {
            characteristics.forEach(characteristic => {
              if (characteristic.uuid === '326a9006-85cb-9195-d9dd-464cfbbae75a') {
                window.coordinates = characteristic;                
                window.coordinates.addEventListener('characteristicvaluechanged', coordinatesChanged);
              }
              if (characteristic.uuid === '326a9001-85cb-9195-d9dd-464cfbbae75a') {
                window.led = characteristic;
              }
              if (characteristic.uuid === '00002a19-0000-1000-8000-00805f9b34fb') {
                window.battery = characteristic;                
                window.battery.addEventListener('characteristicvaluechanged', batteryChanged);
              }
            })
        })
    })
})
}

function batteryChanged(event) {

    let batteryLevel = event.target.value.getUint8(0);
    document.querySelector('#batteryL').textContent = batteryLevel + '%';
    document.querySelector('#batteryIndicator').textContent = batteryLevel + '%';
}

function coordinatesChanged(event) {
  
  let value = event.target.value;
  let a = [];
  // Convert raw data bytes to hex values
  for (let i = 0; i < value.byteLength; i++) {
    a.push(('00' + value.getUint8(i).toString(16)).slice(-2));
  }
  if (a[0] === '13' && a[1] === '05') {
  //  document.querySelector('#valueHEX').textContent = 'x: ' + ' ' + a.join(' ');
}

//QUATERNION  
  if (a[0] === '19' && a[1] === '07') {
    document.querySelector('#valueHEX').textContent = 'Quartenion: ' + ' ' + a.join(' ');
    let w = event.target.value.getFloat32(14, 4, true);
    document.querySelector('#w').textContent = w.toFixed(2);
    let qx = event.target.value.getFloat32(6, 4, true);
    document.querySelector('#qx').textContent = qx.toFixed(2);
    let qy = event.target.value.getFloat32(2, 4, true);
    document.querySelector('#qy').textContent = qy.toFixed(2);
    let qz = event.target.value.getFloat32(10, 4, true);
    document.querySelector('#qz').textContent = qz.toFixed(2);
  }
  
//LINEAR ACCELERATION  
 if (a[0] === '19' && a[1] === '0a') {
  document.querySelector('#valueHEX').textContent = 'Linear Acceleration: ' + ' ' + a.join(' ');
    let y = event.target.value.getFloat32(10, 4, true);
    document.querySelector('#y').textContent = y.toFixed(2);
    let z = event.target.value.getFloat32(2, 4, true);
    document.querySelector('#z').textContent = z.toFixed(2);
    let x = event.target.value.getFloat32(6, 4, true);
    document.querySelector('#x').textContent = x.toFixed(2);

}

//EULER ANGLES
  if (a[0] === '19' && a[1] === '08') {
   document.querySelector('#valueHEX').textContent = 'Euler: ' + ' ' + a.join(' ');
  
  let yaw = event.target.value.getFloat32(14, 2, true);  
  document.querySelector('#yaw').textContent = yaw;

  let roll = event.target.value.getFloat32(12, 2, true);  
  document.querySelector('#roll').textContent = roll;

  let pitch = event.target.value.getFloat32(10, 2, true);  
  document.querySelector('#pitch').textContent = pitch;
}

I'm able to control an object rotation in 3D on screen (embedding three.js in Hype) with Quaternions and it's pretty accurate. But I can't figure out position. I have been reading up how difficult it is. But I'm not striving for too much accuracy, I just want to be able to use the bluetooth device as a basic game controller, which would be an achievement for me. I thought that if I just focus on a 2d implementation first I can aways add another axis later?

I just think it would be cool to play pong with a controller like this. Or asteroids. Then gradually get more sophisticated as I learn more.

Stumbled onto this article while searching and I think it may be promising: https://codepen.io/OliverBalfour/post/implementing-velocity-acceleration-and-friction-on-a-canvas

Thanks

If you take an acceleration magnitude sensor data, add to a velocity, and then have that velocity add to a position, you'll wind up with something.

It isn't clear how well this would work though - first, the sensors would need to be aligned precisely with the screen, or at least you'll need to do some math to make sure the sensor accelerations are converted to screen coordinates properly. Second, there will probably be tons of drift in these sensors. I don't know how long it would hold up. See:

https://www.researchgate.net/post/How_do_we_determine_the_position_of_an_object_with_accelerometer_and_gyroscope_sensor_data

You may want to check with the manufacturer to see if the sensors have a proper way to capture 3D positioning.

The math is beyond me I'm afraid. Not sure how to integrate, never mind integrating twice! But I will try and learn.

My first goal is to visualize the curve from X value with a path so I can see how the value from the accelerometer is mapped. That seems to have worked, but ideally I would remove points I no longer need from the beginning of the path at the same time as new points are pushed to the end so the path doesn't run out of space.

The second goal is to understand the logic of how to apply the data to the object. I have added a Direction display so when the object moves left it should show "left" and when object is moving right it shows "right". Also a zeroCount, because it seems to me that:

  1. If the initial value goes above 0.25, start moving object right.
  2. While it is above 0 keep moving right.
  3. If we cross 0 the first time, increment zeroCount and keep moving right.
  4. While zeroCount = 1, and value is below -0.25, keep moving right.
  5. Only when we come to 0 for the second time, reset zeroCount and start again.

Same for movement to the left, but only if the initial value is less than -0.25 and everything is reversed. That seems kind of logical to me, but doesn't work.

Cheers

  var accelerationValues = [];
  var accelerationPointsArray = [];
  
  var viewBox = document.querySelector('#acceleration_path').parentNode;
  viewBox.setAttribute("viewBox", "0 -50 866 105");
  console.log(viewBox)
  
  var steps = [];
  var step = -1;
  var stepSize = 1;
  var command = 'M ';
  var i = 0;
  var zeroCount = 0;
  
var paddle = document.querySelector('#paddle');
var currentPos = hypeDocument.getElementProperty(paddle, 'left')
var posX = paddle.style.left;

setInterval(function(){
var newX = document.querySelector('#x').textContent;
var zero = document.querySelector('#zero');
var direction = document.querySelector('#direction');

if (!(newX > -0.2 && newX < 0.2) && newX < 0.25 && newX > -0.25){
zeroCount++
zero.textContent = zeroCount;
}

if (newX > 0.25 && zeroCount % 2) {
  hypeDocument.setElementProperty(paddle, 'left', currentPos + newX*20)
  direction.textContent = ' right'
}

if (newX > 0.25 && !(zeroCount % 2)) {
  hypeDocument.setElementProperty(paddle, 'left', currentPos - newX*20)
  direction.textContent = ' left'
}

    step += stepSize;
    steps.push(step);
    accelerationPointsArray.push(command + steps[i] + ' ' + newX*20 + ' ');
    command = 'L ';
    
    var acceleration = document.querySelector('#acceleration_path');
    acceleration.setAttribute('d', accelerationPointsArray.join(' '));
    
    currentPos = hypeDocument.getElementProperty(paddle, 'left')
	i++
	
},20);

The most basic thing you can try is (for each axis):

  • keep a variable for position
  • keep a variable for velocity
  • add the accelerometer data to velocity
  • add the velocity to your position
  • set the position using Hype APIs

You'll need to run this frequently (via setInterval or the like), and it probably won't work right if your polling doesn't sync up with the accelerometer data (I don't know how the device decides to keep/clear its values).

Thanks @jonathan

I think I managed to implement your suggestion and it is kind of working in principle and with some calibration may work better. The acceleration part seems to work, but the direction left right breaks down.

I still think that there is a flaw in the logic in the direction of movement. So some condition needs to be implemented to check which way the value goes when the movement starts.


  var accelerationValues = [];
  var accelerationPointsArray = [];
  
  var viewBox = document.querySelector('#acceleration_path').parentNode;
  viewBox.setAttribute("viewBox", "0 -100 866 200");
  
  var steps = [];
  var step = -1;
  var stepSize = 1;
  var command = 'M ';
  var i = 0;
  var zeroCount = 0;

  var velocityX = 0;
  var newVelocityX = 0;
    
  var accelerationX = 0;
  var newAccelerationX = 0;
  
  var Time = 0;
  var newTime = 0;

  
var paddle = document.querySelector('#paddle');
var currentPos = hypeDocument.getElementProperty(paddle, 'left')

setInterval(function(){

Time = i;
accelerationX = +document.querySelector('#x').textContent / 9.81;
velocityX += ((newAccelerationX + accelerationX) / 2) * (newTime - Time);


var direction = document.querySelector('#direction');

if (newAccelerationX > 0.5) {
 currentPos += ((newVelocityX + velocityX) / 2) * (newTime - Time); 
  direction.textContent = ' right'
}
 
if (newAccelerationX < -0.5) {
 currentPos += ((newVelocityX + velocityX) / 2) * (newTime - Time); 
  direction.textContent = ' left'
}

hypeDocument.setElementProperty(paddle, 'left', currentPos)

    step += stepSize;
    steps.push(step);
    accelerationPointsArray.push(command + steps[i] + ' ' + accelerationX*60 + ' ');
    command = 'L ';
    
    var acceleration = document.querySelector('#acceleration_path');
    acceleration.setAttribute('d', accelerationPointsArray.join(' '));
 
	i++
		
	newAccelerationX = +document.querySelector('#x').textContent;
	newVelocityX = velocityX;
	newTime = Time;
	
},20);

Thanks

Yeah, it seems like the sample rate is probably too small to be accurate enough with the simplest method, so you'd likely need calculus. Even then, it seems that these type of sensors won't do a great job without a lot of drift. You may want to contact the manufacturer and see what they recommend, and if they have any sample code for this (even if in a different programming language). Porting to javascript and then interacting with Hype is probably easy compared to any math required. (At least, this would be my best guess never having tried what you're trying!)

Unfortunately manufacturer is not very good at responding and everything runs via c-sharp and python or via node and feels pretty confusing. So I'm reverse engineering stuff and looking at forums and articles.

I thought that maybe using native physics in Hype would be better, so used a method I found by @Photics here Physics Mini Templates for Tumult Hype 4

It's quite interesting, but still feels like the same logic problem. If the motion is continuous the response is pretty good it seems. But if there is a stop-start, the motion is reversed back to start (or close enough). So I made a little file to illustrate my thinking.

scurve.zip (55.9 KB)

Any suggestions how to make such logic work?

Here is the code I'm using at the moment:

		
var paddle = document.querySelector('#paddle');
var currentPosX = hypeDocument.getElementProperty(paddle, 'left')
var currentPosY = hypeDocument.getElementProperty(paddle, 'top')
		
  var accelerationXPointsArray = [];
  var accelerationYPointsArray = [];
  
  var viewBoxX = document.querySelector('#accelerationX_path').parentNode;
  viewBoxX.setAttribute("viewBox", "0 -100 866 200");
  var viewBoxY = document.querySelector('#accelerationY_path').parentNode;
  viewBoxY.setAttribute("viewBox", "0 -100 866 200");
  
  var steps = [];
  var step = -1;
  var stepSize = 1;
  var command = 'M ';
  var i = 0;
    
  var accelerationX = 0;
  var accelerationY = 0;

function mjsApplyForce (xPosition, yPosition, xForce, yForce) {
hypeDocument.setElementProperty(paddle, 'rotateZ', 0);
Matter.Body.applyForce(hypeDocument.getElementProperty(paddle, "physics-body"), {x: xPosition, y: yPosition}, {x: xForce, y: yForce});
}

setInterval(function(){

accelerationX = document.querySelector('#x').textContent / 9.81;
accelerationY = document.querySelector('#y').textContent / 9.81;

mjsApplyForce(currentPosX, currentPosY, accelerationX, -accelerationY)

    step += stepSize;
    steps.push(step);
    accelerationXPointsArray.push(command + steps[i] + ' ' + accelerationX*60 + ' ');
    accelerationYPointsArray.push(command + steps[i] + ' ' + accelerationY*60 + ' ');
    command = 'L ';
    
    var accelerationXPath = document.querySelector('#accelerationX_path');
    accelerationXPath.setAttribute('d', accelerationXPointsArray.join(' '));
    var accelerationYPath = document.querySelector('#accelerationY_path');
    accelerationYPath.setAttribute('d', accelerationYPointsArray.join(' '));
 
	i++
	
},50);

Thanks

1 Like

I'm not sure what's wrong, so I don't know how to help. However, you might want to know about the Photics-Physics-Bridge… Photics-Physics-Bridge (ppb.js) – Photics.com

That was designed to make it easier to work with Hype's (Matter.js) Physics engine.

2 Likes

Can't find any definitive answers searching the web, so time for some lateral thinking.

Feels to me that the data from accelerometer is pretty similar to a sound wave, in terms of having positive and negative amplitude and it feels similar logic with compression and rarefaction for sound instead of direction for acceleration.

I don't know, perhaps it's a bit of a rabbit hole, but as there are presumably more people working with sound in javascript rather than physics, maybe I can find a concept applicable to my problem with detecting direction.

If nothing else, I may be able to implement a sound waveform visualization but pass acceleration values to it.

If the problem is that the accelerometer is changing too quickly for smooth controls, maybe average it out.

Is that the problem? If so, the closest match that I have right now is the FPS gauge… Free Template Tuesday #6 – Tumult Hype “FPS” – Photics.com …when I used the exact values, the needle would move too erratically, so I used Hype's JavaScript API to slowly transition the needle.

Part II of the FPS template shows how to use an array to collect the FPS data… Free Template Tuesday #8 – Tumult Hype “FPS” (Part II) – Photics.com …so perhaps that technique could be used (or perhaps modified) to smooth out the change in acceleration.

It could also be a limitation of the technology. An accelerometer uses springs… Accelerometer - Wikipedia …at least I think that's how it works. When the movement stops in one direction, that's going to cause the spring to act differently.

It looks like you're having trouble with “oscillations”. Wikipedia puts the word “Wiggles”in parenthesis. Perhaps the accelerometer mass is not dampened enough for your project needs, or perhaps the device is broken?

Around 2010, when the iPhone was still new technology, I was using the accelerometer for game development. There is some information about using the Accelerometer as a game controller in my GameSalad Textbook… GameSalad Textbook – Photics.com …but I'm not sure if it will help you.

Ultimately, I don't remember using accelerometer data much after making that book. :man_shrugging:t2:

1 Like

Thanks @Photics, I had a go at averaging out the inputs, what's interesting is that it makes the issue clearer to understand as it simplifies the graph, so it's more visible.

When the line is at 0 there is no acceleration, when initial peak is above zero move left. When initial trough is below zero, move right. Then you can observe the 'bounce back', which is what I want to get rid off.

The first movement from zero is correct, but then as the values reverse (due to deceleration and coming to a stop) the box bounces back. I need it to just come to a stop.
I don't think the device is faulty or the values are too fast, although filtering will improve things. I just think there needs to be a condition in the logic, that keeps the block moving in the same direction until the second zero crossing after the movement is initialized.

Here is my basic average implementation. 30 is a bit of an arbitrary number for now but good for demonstration.

The value is received "onvaluechanged" so I did this:


window.counter++

    window,avgX += x;
    if (window.counter > 30){
    var averageX = avgX/30;
    document.querySelector('#x').textContent = averageX.toFixed(2);
    window.counter = 0;
    window.avgX = 0;
    }

Thanks