Multi-exit neural networks have recently boomed in edge computing to maximize the computing power of different devices. However, many real-time tasks running on edge computing applications have encountered unpredictable exiting frequently due to system power outages, high-priority preemption, etc., which have been overlooked by multi-exit models until now. To tackle this issue, it is critical to decide at which branch the multiexit model exits so that the unpredictable exit will always come with desirable results. In this paper, we propose EINet, a samplewise planner of real-time multi-exit deep neural networks, which achieves efficient Elastic Inference with unpredictable exit while guaranteeing best-effort accuracy on different edge platforms. Therefore, a given trained deep neural network is first partitioned into multiple blocks with one exit each by EINet. Then EINet obtains the block-wise model profiles, including the block-wise accuracy and inference time. Using the model profiles, EINet is able to dynamically determine which exits to take during the inference task for each sample. We introduce Confidence Score Predictors to dynamically adapt the uniqueness of the input samples, and the Search Engine to efficiently find the near-optimal plan during the elastic inference. EINet is evaluated extensively using multiple DNNs and datasets with unpredictable exits. Results show that EINet can achieve the highest average accuracy compared with multiple baselines.
