NeuralOS: Towards Simulating Operating Systems via Neural Generative Models
Journal:
arXiv
Published Date:
Jul 11, 2025
Abstract
We introduce NeuralOS, a neural framework that simulates graphical user
interfaces (GUIs) of operating systems by directly predicting screen frames in
response to user inputs such as mouse movements, clicks, and keyboard events.
NeuralOS combines a recurrent neural network (RNN), which tracks computer
state, with a diffusion-based neural renderer that generates screen images. The
model is trained on a large-scale dataset of Ubuntu XFCE recordings, which
include both randomly generated interactions and realistic interactions
produced by AI agents. Experiments show that NeuralOS successfully renders
realistic GUI sequences, accurately captures mouse interactions, and reliably
predicts state transitions like application launches. Although modeling
fine-grained keyboard interactions precisely remains challenging, NeuralOS
offers a step toward creating fully adaptive, generative neural interfaces for
future human-computer interaction systems.
