How does the phone know where you moved?

GPS isn't accurate enough indoors, and the phone can't rely on a fixed external tracking system. Instead, Apple's ARKit uses a technique called Visual-Inertial Odometry (VIO) to continuously estimate the device's position and orientation in the real world.

In this article, we'll explore how ARKit combines camera images and motion sensor data to achieve accurate real-time tracking.

About Visual-Inertial Odometry (VIO)

Visual-Inertial Odometry is a technique that estimates the movement of a device by combining:

• Visual information from the camera

• Motion information from inertial sensors

The goal is simple:

Determine where the device is and how it is oriented at every moment.

The term can be broken down into:

• Visual → information from camera images or feature data

• Inertial → information from accelerometers and gyroscopes

• Odometry → estimating motion over time

The camera observes the environment while the motion sensors measure how the phone moves. Together they create a far more reliable tracking system than either could achieve alone.

The Two Main Sources of Information

Camera Data: The Phone's Eyes

ARKit continuously analyzes frames captured by the device camera also known as feature data, (x, y) coordinate within an image. Instead of understanding the entire image like a human would, it focuses on identifying distinctive visual features like contrast, edges and texture-rich regions.

When the phone moves, those visual points appear to shift position in the image. By measuring this movement, ARKit can infer how the device has moved through space.

IMU Data: The Phone's Sense of Motion

The Inertial Measurement Unit (IMU) consists mainly of:

• Accelerometer (measures linear acceleration)

• Gyroscope (measures rotational velocity)

Unlike the camera, which typically captures around 30–60 frames per second, IMU sensors can provide measurements hundreds of times per second.

This makes IMU data extremely fast, available even during rapid movement and useful when camera images become blurry. However, IMU measurements accumulate error over time, causing drift.

Solving the 6 Degrees of Freedom Problem (6DoF)

6DoF is about figuring out where the device is (x, y, z) and figuring out its orientation (pitch, yaw and roll).

ARKit continuously estimates this pose in real time. Every virtual object placed in an AR scene depends on accurate 6DoF tracking. If the pose estimate becomes inaccurate, virtual objects appear to drift, slide, or jump.

How ARKit Uses Sensor Fusion

Sensor fusion is the process of combining information from multiple sensors into a single, more accurate estimate.

ARKit's tracking pipeline roughly works like this:

1. IMU sensors measure motion continuously.

2. ARKit predicts where the device should be next.

3. The camera captures a new image.

4. Visual features are detected and matched.

5. The visual observations are compared against the prediction.

6. Errors are calculated.

7. The pose estimate is corrected.

This cycle repeats many times per second. The result is a stable estimate of the device's position and orientation.

Camera Correction: Fixing IMU Drift

Suppose the gyroscope predicts that the phone rotated by 90°. After several seconds, tiny sensor errors accumulate. The phone may actually have rotated only 88°. This small discrepancy grows over time.

The camera helps detect this error. By recognizing previously observed visual features, ARKit can determine:

"The environment doesn't match the predicted pose."

The system then adjusts the pose estimate to align with the visual observations. This continuous correction prevents drift from becoming noticeable. Without camera correction, virtual objects would slowly wander away from their intended positions.

What makes ARKit remarkable is that all of this happens in real time on a mobile device. Every second, the system processes camera frames, reads hundreds of IMU measurements, tracks visual feature, estimates 6DoF pose, corrects accumulated error and updates the AR scene.

All within milliseconds.

The result is the seamless AR experience users take for granted.

Conclusion

Visual-Inertial Odometry is the foundation of ARKit's world tracking capabilities. The camera provides environmental awareness, the IMU provides fast motion updates, and sensor fusion combines both into a robust tracking system. The constant cycle of prediction and correction is what allows AR experiences to feel stable, accurate, and believable.

The next time you place a virtual object in your room and watch it stay perfectly fixed as you move around, take a moment to think about what is happening behind the scenes. Your device is continuously detecting visual features, processing hundreds of sensor measurements, predicting its motion, correcting accumulated errors, and updating its understanding of the world - all in real time.