Mutually distilled banded spatiotemporal dense model with multi attention for continuous sign language recognition

Abstract

Sign Language Recognition (SLR) systems translate gestures into labels, newlinefacilitating communication between sign language users and spoken language newlineusers. Sign Languages (SL) are visual, with unique grammar, used by about 72 newlinemillion people globally across over 200 distinct languages. SL consists of newlinemanual elements (hand shapes, orientations) and non-manual elements (facial newlineexpressions and body posture). SLR technology leverages advancements in newlinegesture recognition to facilitate more inclusive communication in various newlinesectors, including education, healthcare, and Human-Computer Interaction newline(HCI). newlineSLR distinguishes between static (unchanging gestures) and dynamic newline(continuous movements) sign languages. This work focuses on Continuous Sign newlineLanguage Recognition (CSLR), which translates sign videos representing full newlinesentences into corresponding label sequences, in contrast to Isolated SLR, which newlinedeals with individual words. Challenges in CSLR include understanding newlinespatiotemporal features, motion sequences, varying sign lengths, signer pace, newlineand complex grammar. Real-world deployment of CSLR models is challenging newlinedue to computational complexity, with current research focusing on balancing newlineefficiency and computational costs for deployment on resource-limited devices. newlineFor CSLR to be effective, its spatiotemporal features must be extracted, newlineintegrating spatial characteristics (hand shapes, positions) and temporal elements newline(movement trajectories, transitions). Techniques such as Convolutional Neural newlineNetworks (CNNs), Recurrent Neural Networks (RNNs), and attention newlinemechanisms (spatial, temporal, and channel attention) are crucial for capturing newlinethe nuances of sign language. newline